AI Engineer Assessment: What the Technical Test Checks

An AI engineer assessment is usually a timed LLM knowledge test, a coding task against a model API, and a take-home you defend on a call. Here is what each stage grades, the mistakes that sink candidates, and a five-question self-check.

14 min read

An AI engineer assessment is the technical screen that comes before, or replaces part of, the interview loop for AI engineer roles. It rarely asks you to derive backpropagation or train a model. It checks four things: whether you understand how an LLM actually behaves, whether your code survives a model that returns something unexpected, whether you can build and measure a small retrieval system, and whether you can say what breaks when that system meets real users.

The format varies by company, but most AI engineer technical tests combine some of these five stages:

  1. A timed knowledge test: multiple-choice or short-answer questions on LLM mechanics, usually 30 to 60 minutes on a hosted assessment platform.
  2. A coding task: live or timed, in Python, calling a model API with tools or structured output.
  3. A take-home build: a small RAG or agent system, scoped to four to eight hours.
  4. Evaluation questions: "how do you know it works?", asked about your take-home or as standalone questions.
  5. A review call: an engineer reads your submission with you and asks what happens in production.

If you're earlier in the process and want the full interview loop rather than the assessment, how to prepare for an AI engineer interview covers the five knowledge areas those rounds test. This article is about the test itself: what each stage grades, and where candidates lose points.

Check yourself first

Five questions from squizzu's LLM question bank, one per stage, written in the scenario style that most AI engineer knowledge tests now use. Answer before you read the rest: whichever ones you miss tell you which sections below to read closely.

Squizzu Logo
LLMs • LLM Architecture and Mechanisms

Question 1 / 5

A team ships an LLM coding-and-math assistant and hits three puzzling defects: multi-digit arithmetic is unreliable, prompts behave differently depending on whether they end with a trailing space, and generated Python sometimes has broken indentation. Which underlying cause links all three?

The mapping, in order: the tokenization question belongs to the knowledge test, streaming to the coding task, chunk size to the take-home, metrics to evaluation, and the RAG incident to the review call. If you missed the first one, the knowledge-test section matters most to you, because that stage is often an automatic cut-off before a human reads anything. A miss on the last two usually means the take-home will look fine and the review call will not.

What each stage of an AI engineer assessment grades

Stage Typical length What is graded Where candidates lose points
Knowledge test 30–60 min Precise understanding of LLM mechanics Recognising terms without knowing the mechanism
Coding task 60–90 min Working code that handles bad model output Happy-path code with no validation or retries
Take-home 4–8 h Design decisions and their justification Overbuilding, no measurement, thin README
Evaluation Part of the review A repeatable way to know it works "I tried a few questions and it looked good"
Review call 45–60 min Production judgement No answer for latency, cost or injection

The common thread: every stage after the first is graded on how you handle the case where the model is wrong. A submission that works on the demo inputs and has no answer for the failures scores lower than a simpler one that names its limits.

Stage 1: the timed knowledge test

This is the part most often automated, and the cut-off is usually a score rather than a reviewer's impression. The questions have shifted from definitions ("what does RAG stand for?") to scenarios ("a team sees X; what is the cause?"), which punishes vocabulary memorised without the mechanism behind it.

The topics that come up most, and the trap each one sets:

Topic What the question usually looks like The trap
Tokenization A model misbehaves on numbers, whitespace or another language; find the cause Blaming the model's reasoning when the cause is how the text was split into subword tokens
Context window vs. output limit A reply stops mid-sentence despite a huge context window Treating the two limits as one; the output cap is a separate per-request setting
Sampling Two parameters set at once; which one shapes the output? Knowing what temperature is but not how top-k and top-p truncate the distribution
Prefill and decode Why is the first token slow, or why do output tokens cost more? Missing that the prompt is processed in parallel and the output one token at a time
Base vs. instruction-tuned A model continues the prompt instead of answering it Not recognising a base model that was never trained to follow instructions

Question: What does lowering the temperature actually change?

Short answer: Temperature divides the model's logits before softmax. A low temperature sharpens the distribution toward the most likely tokens, and at zero the API effectively picks the single most likely token each step. It does not make the model more accurate; it makes it more repeatable.

What the interviewer is testing: Whether you separate randomness from correctness. "Set temperature to 0 to stop hallucinations" is the wrong answer they are screening for: a greedy model can be confidently and consistently wrong.

Common follow-up: Why can output still differ between two calls at temperature 0? (Serving-side nondeterminism: batching and floating-point ordering on the provider's GPUs. You get repeatability, not a guarantee.)

Stage 2: the coding task (the "AI coding test")

The usual shape is 60 to 90 minutes in Python: call a model API, give it one or two tools, and return a structured result. Sometimes it is a debugging exercise instead, where you get a broken script and must find why the agent loops or the JSON parse fails.

Graders run your code against inputs you didn't see, and those inputs are chosen to make the model misbehave. What they look for:

  • The tool-calling loop done correctly. The model returns a request to call a function; your code executes it, appends the result to the message list, and calls the model again until it produces a final answer. Know exactly what goes into the messages on each call, because the API keeps no state between them.
  • Retries that are safe to repeat. A timeout does not mean the call failed on the other side. If a tool books, pays or sends, a blind retry does it twice; an idempotency key prevents that.
  • A bounded loop. A maximum number of iterations, so a model that keeps calling the same tool cannot run until the token bill arrives.
  • Basic hygiene. The API key comes from an environment variable, the model name is configurable, and token usage is logged.

Question: The model returns invalid JSON in production about one call in fifty. What do you do?

Short answer: First, make invalid JSON harder to produce: use the provider's structured-output or JSON-schema mode if it has one. Then validate every response against the schema regardless, retry once with the parser's error message in the prompt, and send whatever still fails to a logged fallback path rather than to the user. Track the failure rate as a metric so a model upgrade that changes it is visible.

In code, the minimum a grader wants to see is a validation step and a bounded retry, not a bare json.loads:

from pydantic import BaseModel, ValidationError

class Ticket(BaseModel):
    category: str
    priority: int

def classify(text: str, max_attempts: int = 2) -> Ticket:
    prompt = build_prompt(text)
    for _ in range(max_attempts):
        raw = call_model(prompt)  # your provider's SDK call
        try:
            return Ticket.model_validate_json(raw)
        except ValidationError as err:
            # Show the model what was wrong instead of asking the same question again.
            prompt = build_prompt(text, previous=raw, error=str(err))
    raise ClassificationFailed(text)  # logged and routed to a fallback, never shown raw

What the interviewer is testing: The habit of treating model output as untrusted input. Code that does json.loads(response) and assumes success is the most common reason a coding task fails.

Common follow-up: What is the cost of constrained decoding? (A per-step mask over the vocabulary, which adds some latency, and a schema that is too strict can force the model into a valid but wrong answer.)

Stage 3: the take-home build

The most common prompt is some version of "build a question-answering assistant over these documents" with four to eight hours of expected effort. Agent variants ("triage these support tickets using these tools") are rising, but the grading is the same: reviewers read your decisions more closely than your features.

Three technical choices come up in almost every review. Chunking is a trade-off with a cost on both sides, and reviewers expect you to name both; splitting on document structure (headings, sections) often beats fixed windows. Exact terms such as error codes and product SKUs are where pure vector search fails, which is what hybrid search (vectors plus keyword/BM25) and a reranking step fix. And the amount of context matters: models use information at the start and end of a long prompt more reliably than in the middle, so retrieve wide, rerank, and pass a few chunks rather than twenty.

You don't have to implement every fix. You do have to show you know where your version falls short, and the README is where reviewers look for that. A structure that works:

## What it does
One paragraph. Which questions it answers well, over which documents.

## Design decisions
- Chunking: split on H2 sections, max 400 tokens. Why: answers rarely cross sections.
- Retrieval: vector top-20, reranked to top-4. Skipped hybrid search (see Limitations).

## Evaluation
- 15 questions in eval/golden.jsonl, each with its source document.
- Retrieval recall@4: 13/15. Answer faithfulness (judged, spot-checked by hand): 12/15.
- Run: python eval/run.py

## Limitations
- Misses questions that quote exact error codes (no keyword search).
- One failure traced to a table split across two chunks.

## With another week
Hybrid search, table-aware chunking, a cost-per-query budget.

The numbers in that skeleton are illustrative; yours will differ. What matters is that they exist. The most common way to lose points is overbuilding: an agent framework, three vector stores and a UI, with no measurement of whether any of it answers correctly. A plain pipeline with an evaluation script beats it.

Stage 4: the evaluation questions

Whether asked about your take-home or on their own, these questions decide most senior-versus-junior calls. "How do you know it works?" has a structured answer, and it starts with a golden set: a fixed list of real questions with known sources and expected answers, rerun on every prompt or model change. It is regression testing for behaviour.

The point reviewers probe is that each row is scored twice, once for retrieval and once for generation:

Question Source document Source in top-4? Answer faithful to sources? Diagnosis
How do I rotate an API key? security.md Yes Yes Pass
What does error E7719 mean? errors.md No n/a Retrieval miss: exact code, no keyword search
Can I export data to CSV? export.md Yes No Generation: answered from general knowledge, ignored the doc

Without the split, rows two and three are both just "wrong", and you can't tell whether to fix the retriever or the prompt. The rest of a strong answer:

  • Automated judges, calibrated. An LLM grading outputs scales far past human review, but it has biases toward position and length. Spot-check it against a small human-labelled sample before trusting the numbers.
  • The process, not just the outcome, for agents. An agent can reach the right answer by doing something unacceptable along the way. Scoring the trajectory catches that.
  • Skepticism about public benchmarks. A high leaderboard score is weak evidence for your use case, partly because popular test sets leak into training data. Your own golden set is the number that matters.

Question: How would you evaluate the RAG system you built for the take-home?

Short answer: With a golden set of questions drawn from the documents, each tagged with the document that answers it. Retrieval is scored by whether that document appears in the top-k results. Generation is scored on whether the answer sticks to the retrieved text and addresses the question, using a calibrated LLM judge plus a human check on a sample. The set runs on every change, and regressions block the merge.

What the interviewer is testing: Whether you can turn "it seems to work" into a number. Candidates who describe reading outputs until they "looked right" are signalling that every future prompt change is a gamble.

Common follow-up: Retrieval recall is high but users say answers are wrong. Where do you look? (Generation: the model ignoring or misreading the retrieved context, context truncated by the token budget, or the right chunk buried among many irrelevant ones.)

Stage 5: the review call

An engineer opens your take-home and asks what happens when it meets real traffic. The questions cluster in three areas.

Latency. Know which latency number users actually feel and which one your benchmark measured, because they are not the same. Then the levers: streaming, caching the shared prefix of the prompt (otherwise a long system prompt is recomputed on every request), routing easy queries to a smaller model, and shrinking the prompt itself.

Cost. Output tokens cost more than input tokens because they are generated one at a time. Know roughly what one request to your take-home costs, and which part of the prompt dominates it.

Security. Expect at least one injection question. The core fact is that the model cannot reliably tell instructions you wrote from instructions hidden in a document it retrieved or a web page it read. Prompt wording does not fix that, so a good answer is about the system around the model rather than the prompt. Two related points come up often. Model output passed into HTML or SQL must be encoded or parameterised like any user input. And a system prompt is not a place for secrets, because users can extract it.

Question: How would you cut the latency of your take-home in half?

Short answer: Measure first: split time into retrieval, prefill and decode. Then stream the response, enable prefix caching for the fixed part of the prompt, pass fewer and better-ranked chunks so prefill shrinks, and cap the output length. If that is not enough, a smaller model for the generation step, checked against the golden set to confirm quality held.

What the interviewer is testing: Whether you profile before optimising, and whether you check that the faster version still answers correctly.

Common follow-up: What would you monitor in production? (Time to first token and per-token latency, cost per request, the structured-output failure rate, and a sampled evaluation score that would show quality drifting after a model update.)

If the deadline is close

  • Two days: stage 1 topics in the morning, the tool-calling loop with JSON validation in the afternoon, then rehearse the take-home README structure above.
  • A week: add a small build. Twenty markdown files, chunk, embed, retrieve, ask ten questions, score them. At least two answers will be wrong, and working out why gives you a concrete story for the review call.
  • Already submitted the take-home: prepare for stage 5. Write down your system's three weakest points before the call, because the reviewer has already found them.

Fifteen more questions in the same format

These cover the topics above, from sampling parameters and tool schemas to reranking, trajectory evaluation and output handling. Read them as a checklist: any you can't answer with a mechanism and a trade-off belongs on your revision list.

  1. A prompt engineer sets both top_k = 40 and top_p = 0.9 on a text-generation endpoint and asks which parameter is actually shaping the output at each step. How do the two truncation rules differ, and how do they interact when combined?LLMs • LLM Architecture and Mechanisms
  2. What's the difference between a model's context window and an API's max_tokens parameter?LLMs • LLM Architecture and Mechanisms
  3. An application receives this from an LLM API instead of a prose reply: a structured object naming the function get_weather with the argument city set to "Oslo". In the function-calling pattern, what is supposed to happen next?LLMs • RAG, Agents & Tool Use
  4. An agent keeps calling a search tool with vague queries and malformed date ranges. The tool's current schema is shown below. Which revision most directly improves the agent's calls?LLMs • RAG, Agents & Tool Use
  5. A scheduling agent calls a booking API. One call times out at the network layer, but the booking actually succeeded server-side; the agent retries, and the customer ends up with two meetings. Which design change fixes this class of failure?LLMs • RAG, Agents & Tool Use
  6. An autonomous agent occasionally gets stuck calling the same tool with the same arguments, looping until an operator kills the process and the token bill arrives. Which combination of safeguards addresses this failure mode?LLMs • RAG, Agents & Tool Use
  7. Support engineers search a knowledge base for exact error codes like "E4032" and part numbers, but the dense-vector retriever keeps returning semantically related pages that never contain the literal code. Which change most directly fixes this?LLMs • RAG, Agents & Tool Use
  8. A retrieval pipeline pulls 50 candidate chunks in milliseconds using vector search, but only 5 fit the prompt budget and the best ones are not always ranked first. What stage is typically added between retrieval and generation, and why does it work?LLMs • RAG, Agents & Tool Use
  9. An evaluation shows a RAG system answers correctly far more often when the gold chunk is placed first or last among twenty stuffed chunks than when it sits around position ten. Which documented phenomenon explains this, and what does it imply for prompt assembly?LLMs • RAG, Agents & Tool Use
  10. A team wants to evaluate its RAG system with metrics rather than spot checks, and to tell retrieval failures apart from generation failures. Which metric triad, popularized by frameworks such as RAGAS, targets exactly this?LLMs • RAG, Agents & Tool Use
  11. In an agent benchmark, one agent completes a ticket-refund task correctly, but its trace shows it deleted a customer record and recreated it along the way. Outcome-based scoring marks the run as a pass. What does trajectory evaluation add here?LLMs • RAG, Agents & Tool Use
  12. A newly released model posts a surprisingly high score on a popular public benchmark. A skeptical reviewer notes that the benchmark's test questions have been posted on the public web for years and were very likely swept into the model's training crawl. What is this problem called, and why does it inflate scores?LLMs • LLM Training and Evaluation
  13. A RAG service prepends the same long 2,000-token system prompt to every user request. An engineer enables prefix caching on the serving stack. Where does the resulting speedup come from?LLMs • Inference, Serving & Optimization
  14. A penetration tester examines a web application that takes an LLM's answer and interpolates it directly into the page's HTML, and also concatenates it into a SQL statement used for analytics logging. The model can be influenced by user-supplied and retrieved content. Which vulnerability class does the tester report, and what is the fix?LLMs • LLM Safety & Security
  15. A startup places the line 'Never reveal the partner discount codes listed below' followed by the actual codes into its chatbot's system prompt, and treats this as sufficient access control. During a security review of the bot, what should the reviewer conclude about this design?LLMs • LLM Safety & Security

Practise before the test

Recognising a term and choosing the right answer under a timer are different skills, and the knowledge test only measures the second. Practise LLM questions on squizzu: the same scenario format as the self-check above, with an explanation and an in-depth breakdown for every question, so each miss becomes a topic you can close before the assessment.

If the role is mostly about using coding agents to ship product code, AI coding interviews in 2026 covers the agent loop, context engineering and code review. If your application hasn't reached the assessment yet, how to write a resume for AI screening explains how the screen reads your resume, and the AI/ML resume guide covers the vocabulary AI roles match on. For roles that add a classical ML round, 20 Google AI/ML interview questions covers the fundamentals that round asks about.

Frequently asked questions

What is an AI engineer assessment?

It is the technical screening step for AI engineer roles, sent before or alongside the interview loop. Most combine some of five stages: a timed multiple-choice test on LLM concepts, a live or timed coding task that calls a model API, a take-home project (usually a small RAG or agent system), questions about how you evaluated it, and a review call where an engineer walks through your code and asks what would break in production.

What is on an AI engineer technical test?

Expect LLM mechanics (tokenization, context windows, temperature and top-p sampling), tool or function calling, structured outputs and their failure handling, retrieval-augmented generation (chunking, retrieval, reranking), evaluation methods, and production concerns such as latency, cost and prompt injection. Classical ML theory appears only if the role also covers model training.

How long does an AI engineer take-home assignment take?

Most are scoped to four to eight hours of work, often with a few days to submit. Spending far longer rarely helps: reviewers grade the decisions written in your README and whether you measured the result, not the number of features.

Can I use AI coding assistants during an AI engineer assessment?

It depends on the company, and the instructions usually say. Some explicitly allow tools like Claude Code or Copilot on the take-home and then probe your understanding on the review call; timed knowledge tests are usually closed. If the rules are unclear, ask the recruiter before you start, and be ready to explain every line you submit.

How do I prepare for an AI engineer assessment in a week?

Spend the first two days on LLM mechanics and the function-calling loop, build a small RAG system over twenty documents on days three and four, write a ten-question evaluation set for it on day five, and use the last two days to rehearse explaining your design choices and what you would change for production.

We use cookies

Some cookies are needed to run this site. With your consent we also measure how it is used, so that we can improve it.

Cookie policy

Choose what we may measure. You can change this at any time.

Strictly necessary

Essential for the proper functioning of the website. These cannot be disabled.

Performance and analytics

Help us understand how Squizzu is used, diagnose technical issues and improve the service.

AI Engineer Assessment: What the Technical Test Checks | Squizzu