We (Accidentally) Built a Certification Checker

...and found some certifications that didn’t pass

If you’re preparing to take a certification exam you want to know one thing: For any potential question on the exam, what’s the right answer? For certification exams, the “right answer” is easy to determine...or should be: It's whatever the company/organization providing the certification exam says it is. And, it turns out, that can be a problem.

We recently launched an AI-powered exam preparation site and, as a part of that development, created an AI-driven tool that measures how well a certification provider delivers the ‘right answer’ for any objective in their certification For some certifications, we found that support was…well, the term we adopted was “thin.”

We bring this up because we think certifications are important. If you’re defining a certification, you want to ensure you’ve provided the right answers for all of your certification’s objectives. Defining a certification and failing to provide the support for that certification is an abdication of responsibility.

Answering the Question

As a certification junkie (I’ve never met a certification I didn’t want to have), I’ve been using AI tools  for the last year or so to create practice exams as part of my preparation process. That work has grown into a site that leverages AI tools to provide practice exams to other people.

But if that site wasn’t going to be just AI-slop, it needed to meet some key criteria:

• All the exam questions had to be driven the certification exam’s objectives
• All the answers had to be the “right answer” – the answer the provider said was right

That second goal took a bunch of potential certifications off the table: The site couldn’t support any certification where the official documentation wasn’t publicly available (e.g. where the material was behind a paywall, only available in books from the provider, required attendance at approved classes, etc.).

For the remaining certifications, in order to meet our second criteria, the site needed to be able to point to the “right answer” for any objective in the provider’s documentation. In meeting that goal, we inadvertently created a “certification assessment tool” – a tool that told us, for any question for any objective in a certification, if the provider had provided a right answer.

The Problem

We first found that some objectives on some certifications had very few right answers. You've probably bumped into this if you've taken more than one practice exam in a certification: You see questions that revolve around the same one or two right answers cropping up again and again. We classified those objectives as “thin”: It turns out that, for some objectives, there just aren’t a lot of questions you can ask.

But we also found there were some certifications where there was no right answer at all: There was nothing from the provider that said, “For this objective, this is what you should know.” Not surprisingly, we ran into the most issues in this area with two very recent certifications, Anthropic’s developer and architect certifications.

Don’t misunderstand me: I’m not criticizing the Anthropic certifications. I think the certifications are well-designed – the objectives align with what practitioners need to know. To put it another way: If you have the knowledge those objectives identify and can put that knowledge into action, you will be a very good Claude architect/developer. I don’t say that idly: I’ve earned three of Anthropic’s certifications and used the related knowledge in building Claude-based applications. In my opinion, the Anthropic certifications don’t just have good objectives, they have the right objectives.

It’s just that, for some of the objectives in Anthropic’s certifications, our site couldn’t point to a right answer in the Anthropic documentation (and we love Claude very much and looked really, really hard to find answers for those objectives). Fortunately, for people seeking Anthropic’s certifications, several sites have stepped in to fill that gap: AI Cert Guides and Cred Farmer among, I’m sure, others.

It’s just that…well, neither of those site’s answers are official answers. I’m not saying those sites don’t have great answers (personally, I think they both do excellent jobs). But if Anthropic, in marking your exam, disagreed with either of those two sites, you couldn’t claim that you were right and Anthropic was wrong. And that was a problem for our site.

Our Decision

So what do you with certifications that have thin objectives? In our case, we decided to support some of those certifications…to a certain extent, and with qualifiers.

There were some certifications where we felt we could ignore one or (at the most) two objectives without significantly impacting certification exam scores. We decided to support those certifications but not include questions around those objectives in our exams. Effectively, we decided that not all objectives are equally important and, in some cases, the right answer would be based on ‘industry experience’ rather than the provider’s specific technology. We also flag those objectives in the study guide we provide for any certification we support so that users have know what they're getting (or not getting).

For some certifications, though, we decided that our process just couldn’t provide enough right answers for enough objectives for our site to be useful for someone taking the exam. We chose not to support those certifications. And we think that’s too bad because those certifications are often for technologies that we like very much and think are important. It's just that we think the real “right answer” in this case is for certification providers to first identify and then fill those gaps from their documentation (and we re-check those certifications every month, just in case).

It's not a perfect answer but it's one we can live with.