Incremental code deployment to a large cluster in the cloud

Question

How can I do incremental deployment of my code to a large cluster in the cloud? I mean if I have a cluster of say 300 nodes and I want that cluster to get updated with my latest build code but only on the files that are changed (binary and data files), how can I go about it?

I came across murder but it doesn't seem to do incremental file deployment and copies everything down to every node.

I also looked at chef but that does not seem to be a good fit for incremental total code deployment too and it is more suitable to run your installer on every machine and make sure the installed code config files are up to date according to the chef server. It does not look like to bother with the version of the installed program. So I see chef have file control but maybe make sense to use it for a config file but not sure if it makes sense for a code project that has hundred of binaries and data files.

I can sure write my own system to do this but I am wondering if I have a better option for large cluster incremental deployment especially during the development phase?

In my case, the servers are windows and cloud is aws.

Answer 1

Have a look at kwatee (I'm affiliated). It is a lightweight and fully free tool. Among other things it automatically does incremental deployments by comparing the hash of each file against that of what is deployed.