AI Engineer — searchable YouTube transcripts & AI Q&A

Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

Model choice has a negligible effect on engineering output, while individual skill, usage volume, and team environment matter far more, according to Jellyfish's data. For example, a natural experiment where Cursor promoted Grok showed no change in merged PRs per engineer, and switching models or tools generally yields little difference. The biggest lever is the individual engineer, followed by company and team factors, so leaders should focus on improving skills and culture rather than chasing better AI models.

1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

AI failures are increasing exponentially, and analyzing them reveals that deployment controls are more critical to preventing harm than technical model strength alone. The speakers propose a risk-based evaluation framework that decomposes risk into four components—failure frequency, reach, detection, and severity—and advocate for failure-driven testing that actively tries to break the system, as illustrated by an airline chatbot that gave incorrect fare advice, leading to legal liability. They argue that evaluations should cover the entire deployment system, not just the model, and should prioritize high-risk use cases based on materiality.

How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

The speaker argues that when no benchmark exists for an AI agent, you can create your own by inverting the problem: generate test inputs from known answers and check if the agent finds them. He illustrates this with a RAG system where questions were generated from documents, and later with a contract analysis agent where rules and templates were used to create test cases. He emphasizes that this approach allows for rapid iteration and that LLM-as-a-judge is often too noisy to be useful.

Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI

AI-generated writing remains sloppy because models overuse generic words, lack specificity, and fail to adopt a consistent tone, argues Louis-François Bouchard. To address this, he introduces ToneBench, a benchmark that measures how well models can mimic a specific author's voice using detailed prompts and LLM judges. He advises using long, detailed prompts with examples and negative examples, and suggests generating multiple drafts with a cheaper model and having a smarter judge merge the best parts, while noting that human review is still essential.

The AI Content System Behind 200M Views — Sarah Chieng, Cerebras & Cova Lee, OpenAI

Sarah Chieng and Cova Lee argue that AI tools can systematize content creation, but creators must maintain their own voice and taste. They recommend building a swipe file of references, using AI for ideation and scripting, and automating editing and publishing tasks. The key is to use AI to handle tedious work while focusing on creative direction and personal perspective.