David Develops (Software)

Fear Of Deployment

When you start working on a new project there are four really important pieces of information you should be aware of regarding the deployment of your software

1. Are you unafraid of deploying your code to production?
2. Do you know how your code will react in your production environment with production configuration?
3. Does your code react in the same way in your production environment as it does in development?
4. Are you confident in the repeatability of your installation procedure?

Fear and uncertainty in the deployment of software projects is a common theme I’ve seen across all the companies where I’ve worked and it’s never good for your project.  The fear of deployment is always a good indicator of the confidence your staff have in the project they are working on.

A project with unknown qualities and suspicious (or known but unreported) bugs often causes an serious fear of deployment.  There’s nothing like the fear of deploying an application and reducing a business critical system to it’s knees.

I’ve worked on a handful of projects where I’ve taken it upon myself to reign in this deployment fear and chaos, and more often than not, the fear has it’s roots in an uncontrolled development environment.

Configuring A Development Environment

I have very few suggestions regarding which tools you should use to develop your software.  I think that you gain the maximum benefit by allowing developers to pick and choose the absolute best tools for the job, be that technically or by preference.  Allowing developers to use their tools of choice makes happy developers.

By contrast however, the one thing I’m really certain about is that every development environment should start with an absolute base configuration that should not be changed.  This is the first way to solve the eternal “works on my machine” developer problem.  In essence, if all the development environments start off the same, there’s no excuse for a “works on my machine” error.  Not only will this reduce friction between team members, but it’ll speed up the development process because you’re all working to one known configuration.

Furthermore, the development environments alone shouldn’t share this configuration, they should share the base configuration with your production servers.  A huge percentage of software release issues are relating to the differences between production and test and development and the only way to solve what amounts to human error is to make this discrepancy go away.

In the current IT landscape where virtualisation is painfully simple (and often free) there are no excuses not to have a development environment (or at the very least development staging virtual machine) that accurately and completely mirrors your production server.  If you’re struggling with licenses of third party software, produce mocks / proxies / approximations.  I can assure you that the time you spend building these glorified testing rigs will work healthily towards eradicating development-to-production errors.

If It’s Worth Doing Once, It’s Worth Doing Three Times

Another nasty pattern I see frequently is the development-to-production pattern.  If you’re compiling software on your development machines and releasing it to production, “you’re doing it wrong”.  What’s worse than a “works-on-my-machine” conversation with a colleague?  You got it, a “works-on-my-machine” conversation with a system administrator.  It’s not an excuse.  Stop it.

You need to start dog-fooding your deployments, and this means environment cloning and real test servers.  I always follow a simple pattern; developer machine -> developer test box -> system-test box -> user acceptance test box -> production.

In order to do this you’ll need a series of environments (physical or virtual)

• Developer machine
• Developer environment
• Test environment
• UAT environment
• Production environment

That might sound like a lot of hardware but it really isn’t.  Use virtualisation and use it well.  Clone your virtual environments if need be.  Use snapshots and rollbacks to test deployments.  It shouldn’t take any time to administer (and I’ll explain why in the next section) and by the time your code is running on production you’ll have moved it through three different environments.  This is absolutely key to having faith in your deployment procedures.  When you’ve deployed your code so many times that you’re sure it works, in an environment that’s a mirror image of production, production is a far less scary place to be deploying to.

The key to this technique is that the environments all have very clearly defined purposes.  The developer produces his code changes to a system on his development box.  The code seems to work so it’s deployed to the developer environment and checked into your source control repository.  The development environment should be treated as a place where things could potentially get messed up, but it’s also a good place to host joint resources (a database, some middleware) that all the developers make use of.  If the code works in the developer environment it can be promoted to the test environment for the system tester (or a developer wearing a system testers hat) to test.  Once passed on the test box, the code changes can be promoted to the UAT environment for users to preview coming code releases.  Once these changes have been accepted, the code is released into production.

Beware of release insanity.  It makes perfect sense to frequently deploy to the developer environment but only occasionally promote code up to test and UAT.  This should fit around your way of working, just be sure to keep the conceptual integrity of the environments.  If you find a bug in test, don’t fix it in the test environment, fix it in development. It’s up to you to ensure you know what code is running across your environments to avoid any nasty surprises.

One Install To Rule Them All

At this point you’re probably thinking that this all sounds like a huge amount of effort but it isn’t.  The reason it isn’t is that you should, for the most part, be able to deploy your entire system in about 5 clicks / commands.  Don’t let your deployment mechanism be the week link in your application.  There are tonnes of install technologies on the market, both free and costly, to match whichever platform and tool chain you’re working in.

I wish I could remember where I read it, but one of the pieces of setup advice I read long ago that always stuck with me is that you should aim for a one-click install.  You might not achieve it, but you should aim to get very very close.

You need to invest time in configuration management for your application.  Build tools to auto-generate configurations for your deployment environments, build tools to automate setup and deployment.  These tools are worthwhile, because from the moment they’re built, you never need to worry about deploying your software.  They remove risk, and anything that removes risk from software development should be embraced.

Once you’ve build these tools and installers you’ll find your able to do things that you only thought were possible in large companies with seemingly limitless resources.  You’ll be able to configure nightly builds using a continuous integration package of your choice and have the nightly builds install themselves in your developer environment,  You’ll be confident that at any point you can provide a working copy of your software on request.

No Fear Of The Unexpected

The wonderful benefit you get at the end of this process is confidence in your software.

So you’ve released your system to production, do you then feel the need to slavishly test the system in the live environment? 

Imagine that urge going away, knowing that by the time the code hits production that any environmental glitches that could manifest have had the ability to fail three times already.  Any amount of cursory testing you could perform at this point should pale in comparison to the testing you did in the test environment and the user testing environment.  If your “at a glance” testing somehow discovers a bug, you’ve done something very very wrong in the lead up to release.  You should be confident in the fact that your code has been run exhaustively and often by the time it’s reached production.

The world isn’t perfect however, and it’s obviously advisable to a quick audit of the software once installed, just don’t force yourself to repeat your system tests in production (an obvious anti-pattern) and you’ll gain no benefits from doing so.

If you can devise an automated tool to verify your deployments after install, you’ll be able to round off your sense of deployment-security.

A Zen-like State Of Satisfaction

I opened this piece with a few simple questions.

• Are you unafraid of deploying your code to production?
• Do you know how your code will react in your production environment with production configuration?
• Does your code react in the same way in your production environment as it does in development?
• Are you confident in the repeatability of your installation procedure?

I can answer yes to all three questions because I understand my environments and I hope you can too.

[NOTE: This post relates to an old version of MobileTFL - Please see http://www.davidwhitney.co.uk/software for the latest version]

I’ve just compiled what I hope is the final version of this application barring London growing extra underground lines.

I’ve written a small Windows Mobile application for phones running WinMo 5.0+ with Compact Framework 2.0+.  It syndicates the Transport for London live data in similar way to the pre-existing iPhone and Android applications allowing you to view the current tube network status, planned engineering outages and local station notifications at the push of a finger on your Windows Mobile.

It’s currently living at a holding page at http://www.davidwhitney.co.uk/software (soon to become a larger site) and it’s a free download for anyone that uses Windows Mobile.

The 1.0.0.4 update is a reaction to some feedback I’ve received over at http://www.xda-developers.com and comes complete with coloured glyphs representing each tube line and enhanced resolution support for devices with “unusual” resolutions such as the Sony X1.  I’ve also had a stab at some font and UI element scaling so hopefully the interface will look quite natural on all your devices.

I’ve only managed to test it in the wild on a Touch Diamond, Touch Cruise and the Windows Mobile 5.0 Virtual Machine (along with in Windows).  It seems to run admirably in those conditions.

Obviously it requires a data plan so beware of any provider costs.

You can download the .cab directly at this link: http://www.davidwhitney.co.uk/software/repository/MobileTFL/1.0.0.4/MobileTFL.Setup.CAB

A very quick late night post.

I’m a big fan of HTC’s Touch Diamond once you remove all the cruft that the major phone networks like to cram onto their devices.  It’s a pocket sized, powerful smart phone.  I picked one up only a few months ago as a replacement for my HTC Touch, which was starting to feel very very slow compared to some of the handsets on the market, and the addition of HDSPA on the Diamond was the clincher.

Anyway, as a result the good lady has been quite impressed with the functionality of the Diamond.  She works in the media and the ability to access the internet with an almost desktop like experience (and without an iPhone) was very appealing.  While we were in town this weekend sorting out a new contract for her, I noticed that Google’s G1 ships with a London Tube service status application.  Seeing as Eleanor lives in London it seemed like an instantly cool thing to have on hand, but to my surprise, there isn’t anything comparable available for Windows Mobile (or there is and it’s too difficult to track down).

So I wrote one.

I’ve put up a really REALLY retro holding page at http://www.davidwhitney.co.uk/software for the purpose of releasing this application, hopefully over the coming months I’ll use it as a gather place for all the applications I’ve written, both free and pay-for, but for the moment, feel free to go and pick up a copy of the ingeniously named MobileTFL (after the transport-for-London website, where the application sources its data).

If you’re too lazy to click through one link:

Download Mobile TFL for Windows Mobile Here (Cab file)

You’ll have to forgive the exceptionally low-fi website and hilarious low res Visual Studio 2008 virtual machine screenshot.  I’ve only really tested the app on the Diamond I have sitting on my desk right now and it works a treat.

Obviously it requires you have a data plan that’s from a company that believes in Mobile internet rather than customer robbery but it only uses a tiny amount of data (it’s just a Http request).

It also requires the Compact Framework.  I’ve built it under CF3.5 but I suspect it’ll run just fine under CF2.0 (they’re binary compatible after all).

Feedback is more than welcome, just send me an email about it.  I’m especially interested on how the rendering looks on your devices, as the rendering code was written quite quickly and as soon as it looked ok in the VM and on my device I pretty much packaged it up.

I doubt it’ll brick your device, but if it does, you know the drill, it’s your problem.

Time for sleep now.

[Update: Microupdate to version 1.0.0.3 to fix a few bugs and aestetics, the above link has changed, revisit it for the new version]

It occurs to me that I wasn’t thinking straight when I wrote the previous entry regarding my reluctance to use .Net 3.5 SP1 in ‘Encounters’ for hybrid networking.

In a clearer state of mind, it occurs to me that because only the Windows version of the game is going to use the WCF networking implementation, that only the Windows version of the game will require the 3.5 SP1 version of the framework as a result.

The Xbox 360 appears to use a version of the Compact Framework 3.5, however it’s worth noting that 3.5 is a binary compatible set of additional assemblies that run in the .Net 2.0 environment.

What this effectively means is that I can use functionality in 3.5 SP1 (specifically the ability to serailize objects that aren’t marked as DataContracts or Seralizable) for the Windows version of the networking stack without contaminating the code of the game and preventing it from running on a 360.

Even better, it means I don’t need to mark-up my data model with any kind of attributes to support network play on Windows and I shouldn’t need to use a dubious set of byte array wrappers for data that can be typesafe, effectively letting me maintain a “purer” game model.

I’ll move my development environment to 3.5 SP1 and test this theory later, but I suspect I can simplify the networking stack on windows without the need for any messy hacks because of SP1.

Sometimes sleeping on a problem really is the answer; I was about to do something pretty stupid.

I’ve jumped right in to the deep end with my game project (which is going under the working title of “Encounters” – I needed to call the solution something!) and one of the core design goals of the project is the multiplayer focus of the game.

I’m a huge fan of Id Software.  John Carmack is probably my favourite “celebrity computer programmer”.  He’s incredible smart and consistent and has made some of both the most influential and my own favourite games.

Standing on the shoulders of giants - The “Quake” model

I’ve always thought the “Quake model” of single player and multiplayer game development to be a good one.  For those that don’t know, one of the simple design tenants of the Quake engine is that everything is a multiplayer game.  When you play the campaign in the original Quake what the game actually does is start a local game server which you then connect to.  Due to the proliferation of the Quake engine and other engines that have been inspired by it’s design this became quite a common way to build first person shooters with multiplayer support from the offset.  It reduced the implementation of multiplayer to simply having a second player connect to the active session.

Standing on the shoulders of giants, I’ve decided that this is the model I wish to follow for Encounters, especially seeing as the primary work-in-progress game design desires 4-player coop as the main campaign (with NPC assistants if you play with fewer humans).  Because of the way supporting multiplayer from the offset effectively means that all the games “thinking” is done in the server component (collision detection, cheat prevention, state management..) I decide that it’d be the best place to start the implementation.

Prototyping

So far so good, I spent an hour or two last night designing a simple state managing game server and retrofitting it to my previous prototype (an arena with a player-controlled unit moving around it) and moving all the logic and validation to the server side.

At this point the “game server” was a singleton class that the game accessed though a fake “proxy” class (left empty for eventual network implementation) and it worked pretty well.  The collision detection worked on the “server side” when the server was running in the same application domain and as a proof of concept everything was quite sound. I’m still left facing a few issues surrounding the frequency of syncing with the server and client side prediction, but they’re all relatively well solved problems in gaming (and there’s certainly some prior art to take inspiration from in XNA tutorials around the web).

Building a network stack

Once the proof of concept was working I started looking into the networking support provided by XNA and hit the mother of all roadblocks.  Because I’m targeting both the 360 and Windows, networking becomes significantly more complicated.  See, Microsoft offer no direct network access on the 360, via XNA, or even to their licensed partners (internet hearsay claims).  They provide Xbox Live APIs as part of the XNA framework however, which seemed like a decent solution until I realised that these networking APIs function only on a 360 as part of Xbox live.

Not so multiplatform really

So I’ve started thinking around the problem.  I do a lot of work with WCF in my day-job writing distributed systems so my obvious inclination was to provide a WCF client-server architecture for the Windows targeted version of the game.  This adds an additional set of issues.  For a start it looks as though the WCF assemblies (System.ServiceModel) are probably not available on the 360.  You’d think that’s not much of a problem if you’re going to use the Live APIs, but then you realise that if you’re using a version of .Net 3.5 prior to the first service pack, you need to annotate any classes you want to transfer via WCF with the DataContract attribute.  Which is in the System.ServiceModel namespace.

I’m not sure (as I currently don’t have a subscription to deploy my test code onto my 360) what version of the framework the 360 is running, but I’d hazard a guess that it’s pre SP1.  I’m going to have to further research this problem, because if you can use SP1 on the 360 (which supports serialization without any attributes on the classes) then we’re home free.

Multiplatform networking code using Inversion of Control

In the interim I’ve devised a cunning solution (or perhaps workaround, I’m not sure).  See, all of these issues lead me to need two entirely separate network stacks for my game.  One that supported the Windows, and the other that supported Xbox live, without contaminating my data model with mark-up that’s useable in only one or the other of the stacks.

Castle Windsor to the rescue.

This is purely speculative as I haven’t attempted to implement the 360 network stack yet, but I intend to use the Castle Windsor IoC (inversion of control) container to load a separate network stack at runtime, depending on the platform.

My idea is that the game knows about two interfaces IGameServer and IGameServerConnection.  I then create two entirely separate assemblies, one with the networking code for the PC, the other with the Live networking code.  The PC version contains a class called WcfGameServer which implements IGameServer (I’ve already written this code), and a class called WcfGameServerConnection which implementes IGameServerConnection (and acts as a hand crafted WCF proxy).  The 360 implementation will feature XBL counterparts to these classes.

As far as the game is concerned, it’ll use the Windsor container to load an instance of IGameServer, call the StartListening(); method, then use Windsor to load an instance of IGameServerConnection() and call JoinGame() on that connection.  This way, the specific networking implementation is entirely removed from the game and hidden behind these two simple interfaces.

I’ve currently got a good way through implementing the WCF version of this model though I’ve hit a few snags on the way.  Because I’m deliberately developing in a pre-SP1 environment for the sake of this exercise, WCF doesn’t like serializing the game model to send over the wire.  As a result I’ve had to produce an inelegant hack to work around my desire to keep the System.ServiceModel assembly reference clear of my game model.

Trying to keep it light

I’ve marked up my model with Serializable attributes (.NET 1.1, that’ll be fine) and inside my WCF implementation of IGameServerConnection I’m marshalling all my data into byte[]’s before sending it over the wire.  This isn’t ideal as it requires the WCF implementation to manually deserialize the byte[]’s into their correct data types in the service implementation and the client library, but it does work.  Unfortunately at the moment these byte arrays are being stored as XML before being sent over the wire (as is the default WCF way) so I’m going to need to force WCF to binary serialize all it’s data rather than bloat my packets (and as a result, the game latency).  I’m using Net.Tcp so it’s pretty lightweight as far as protocols go but I suspect I’ll need to do some additional fine tuning to make the WCF implementation viable.

Either way, I’ve got a good feeling about the model, subtle hacks aside, and I think this is quite a good way to target both platforms with minimal impact to your game code.

So this is day 3 of development (I’m sure I’m going to loose count pretty quickly).

[Footnote]

It occurs to me that I got a few fundamental things wrong when this post was originally written regarding the requirements for .Net 3.5 SP1.  These mistakes almost complicated the design of the networking stack.  Read more in my follow up here.

It was always a matter of time before my addiction to games and my addiction to programming eventually collided in a horrible mistake.

I’ve decided to start writing a game, using a home grown game engine hopefully with “a little help from my friends”.

I won’t spoil anything too much (as I don’t really want to play my hand too early) but the gist is a 2d, top down, multiplayer action game, with heavy RPG tendencies.  Think of it as a cross between 2d Zelda games, Cannon Fodder and high end raiding in World of Warcraft.  The game design is up in the air at the moment, but hopefully it’ll be derived from a system my good friend Matt has started fleshing out (with the potential for a more RPG leaned game mode, and a more arcade oriented game mode).

The only problem is, I’ve never attempted to write a game before.

So hopefully we can learn together.  I have a rough hit-list of the ground I’m going to have to cover to pull this off, and a rough idea of the technology involved (partially from my other programming pursuits, partially from my enthusiastic following of the games industry) but actually implementing it from the ground up is going to prove an entertaining learning experience.

I’ll tag these posts so people can avoid or watch them at their leisure- there’s a good chance I’ll be dropping into lots of technical detail with complete disregard for the experience levels of my audience. That said, I’m going to be learning a lot of this stuff from scratch (collision detection, graphics rendering, effective ways to script games) while dragging in my bread and butter (distributed systems, C#, etc) to hopefully fill in around the edges.

I’m going to be working in the XNA framework, partially to lower my barrier to entry (seeing as .Net is the day-job and the hobby) and partially because I enjoy the ability to target both the PC and the Xbox360 platform using roughly the same code.

It’s going to be a rocky ride I’m sure, I’ve been working on proof-of-concept code for about a week on and off and it looks like it’s something you can stumble in to (especially with some of the fantastic guides available online).  I’m more than willing to take pointers or tips, so if anyone knows of any especially trustworthy resources on authoring 2d games I’d love to hear some suggestions.