AWS OpsWorks for Chef Automate and Chef version

tl;dr - there seems to be a bug using Chef 14 to bootstrap nodes with the current AWS OpsWorks for Chef Automate, using Chef 13 works as expected.

I’ve started working with AWS OpsWorks for Chef Automate at $WORK and ran into an annoying issue yesterday with Chef 14.x support that I couldn’t find referenced anywhere when doing extensive Google searches.

We’re using Chef somewhat differently than recommended at $WORK because we rolled a our own build pipeline before Chef Automate existed, and that includes dynamically updating build environment versions based on what’s in the repo. So instead of installing the ChefDK and then having to manage Chef versions manually, we usually just drop in a Gemfile with the versions we want, do a bundle install, and then run all the various chef commands using bundle exec .... This lets us quickly and easily switch between Chef versions and other dependencies.

Because this is a new Chef setup I was trying to use the most current version of the Chef gem, which is 14.x. Unfortunately the AWS documentation for supported Chef versions for OpsWorks is pretty out of date, it only mentions Chef 12.x which was end of life as of April 2018. Some other AWS blog posts and the starter kit reference Chef 13.x, but I figured I’d try Chef 14.x since it’s been the main supported version also since April 2018, so shouldn’t be bleeding edge anymore.

Everything worked fine for basic communication with the Chef server – I was able to pull basic information down, push up cookbooks, and otherwise do all normal operations with bundle exec knife.

However, when I tried to bootstrap a new node I’d just get a Doing old style registration with the validation key at ... message – strange both because I have no validation key (they’re not used anymore), and it’s not showing the path to one as expected with that particular message – and then a generic Ruby error ERROR: TypeError: no implicit conversion of nil into String. Turning on knife debug didn’t show anything useful, and many Google searches didn’t turn up anything useful.

The only way I could keep moving forward was downgrading to the Chef 13.x gem instead of 14.x. I looked through the Chef changelog and I don’t see anything specific about knife parameters or bootstrap changing, and nobody else seems to be reporting this issue. Either it’s specific to OpsWorks, or not many people are bootstrapping nodes using 14.x yet with Chef Automate. Perhaps they’re bootstrapping with 13.x and then upgrading each node later to 14.x (there are blogs about doing that upgrade). Or possibly there’s something wonky in my environment, but it seems everything works exactly as I expected after switching to Chef 13.x.