Release notes used to be one of those tasks everyone agreed was important, but nobody really owned. We’d ship a release, someone would scramble to collect changes from developers, and eventually a Markdown file or sometimes an Excel sheet would appear usually incomplete, sometimes outdated, and almost always inconsistent in tone and structure.
A few months ago, I got tired of that loop and built an automated release note generation system. It’s now part of our release workflow, and more importantly, it gave QA a much more reliable way to verify changes.
- The Problem
- The Idea
- The Solution
- How It Works
- The Prompt We Use
- Impact
- Lessons Learned
- Closing Thoughts
The Problem
Before automation, release notes were assembled manually. The process looked roughly like this:
- Developers would drop notes in Slack, PR descriptions, or sometimes directly into Excel
- Someone (usually from QA or a PM) would try to consolidate them
- Important changes were sometimes missed
- Technical changes were either too vague or too detailed
- There was no consistent structure across releases
The biggest issue wasn’t just the time it was lack of reliability.
QA engineers often had to double-check commits anyway because:
- Some changes weren’t documented
- The impact of changes wasn’t clear
- Dependencies between components weren’t not visible
So instead of helping QA, release notes became something they couldn’t fully trust.
The Idea
Instead of asking humans to summarize changes after the fact, we shifted the source of truth:
Let the system generate release notes directly from version differences.
That meant:
- Comparing actual code changes between releases
- Using commit history and PR data as inputs
- Letting AI handle summarization and structuring
This removed the dependency on manual input and made the process deterministic.
The Solution
We built an automated pipeline that generates release notes by comparing:
- The current release branch/tag
- The previous release tag
From that diff, the system collects:
- Commits
- Pull requests
- Code changes
- PR descriptions and comments
Then an AI agent (Devin) processes that data and generates:
- Structured release notes
- Categorized changes (features, fixes, etc.)
- Risk analysis for QA
- Output in both English and Japanese
Everything is posted back into Slack as a .md file.
How It Works
Here’s the workflow we run in production.
1. Release branch creation
We follow a simple convention:
release/{version}
This becomes the anchor point for the release.
2. Tag generation
A release tag (e.g., v2.3.0) is created.
This allows us to:
- Identify the current version
- Locate the previous version tag
- Establish a clear comparison range
3. Slack trigger
Once the release tag is created, a GitHub Action sends a request to a Slack webhook that’s wired into a Slack workflow.
That workflow posts a message in Slack and tags @devin with the playbook macro, which then kicks off the automation.
Additionally, we kept a manual fallback anyone can trigger the same process by tagging @devin with the playbook macro directly in Slack. This is useful in cases where we want to regenerate notes or run it outside the standard release flow.
This setup keeps everything inside the team’s existing workflow while removing any manual steps no one needs to remember to run scripts or trigger jobs separately.
@Devin
!generate_bilingual_release_notes !deep
=========================
Repository : example-repo
Release branch : release/v17.3
=========================
4. Devin AI runs the playbook
Devin executes a predefined prompt that drives the entire process.
It: - Parses version information - Finds the previous release - Analyzes commits, PRs, and diffs - Combines intent (PR descriptions) with actual code changes - Generates structured, bilingual release notes - Performs risk and dependency analysis
5. Version diff analysis
This is the core step.
The system:
- Lists all commits between two versions
- Maps commits to PRs
- Extracts relevant file changes
- Identifies impacted modules
This ensures the output is based on what actually changed not what someone remembered.
6. Release note generation
The output is a structured Markdown file with:
- Summary
- Features / Enhancements / Bug Fixes / Technical Improvements
- PR list
- Risk & dependency analysis
It also includes:
- English version
- Japanese version (collapsible section)
7. Slack delivery
The generated .md file is posted back into the same Slack thread.
This keeps the entire release conversation in one place and easy to track.
The Prompt We Use
This is the core prompt that powers the system. It’s doing most of the orchestration work:
# TASK: Generate Bilingual Release Notes File with Risk Analysis ## 1. ROLE & GOAL You are an expert technical writer and AI development agent with the ability to read and understand code. Your goal is to autonomously generate comprehensive **bilingual (English and Japanese)** release notes, complete with a detailed **risk and dependency analysis** for the testing team, and output the result as a complete Markdown file. ## 2. CONTEXT & ASSUMPTIONS - **Input:** You will be given the repository name and the current release branch name. - **Language Output:** English and Japanese. - **Final Output:** A single Markdown (`.md`) file. - **Audiences:** - **Primary (main body):** Product managers and stakeholders. - **Secondary (final section):** The Testing Team. - **Project Structure Assumptions:** - Release branches follow the pattern: `release/{version}`. - Releases are marked with git tags that follow semantic versioning (e.g., `v2.0.0` or `2.0.0`). ## 3. INSTRUCTIONS Follow these steps precisely: 1. **Initial Analysis:** - The **Project Name** is the `REPOSITORY_NAME` provided as input. - The **Current Release Branch** is the `RELEASE_BRANCH_NAME` provided as input. - verify above name is actually match with repository name as project name before writing to output file. 2. **Determine Versions:** - From the Current Release Branch name, parse the **Current Version**. - Scan the repository's git tags to find the **Previous Version** tag immediately preceding the Current Version. This is your comparison starting point. 3. **Synthesize Change Information (Crucial Step):** - To build a complete understanding of each change, analyze three sources between the Previous Version and the Current Release Branch: i. **Commit Messages & PR Descriptions:** To understand the developer's *intent*. ii. **Actual Code Diffs:** To understand what was *actually* implemented. iii. **Synthesis:** Combine intent with ground truth. While synthesizing, pay close attention to inter-module dependencies and the potential blast radius of changes to inform the risk assessment. 4. **Collate Pull Requests:** - Gather a complete list of all Pull Request titles and numbers that were merged between the **Previous Version** and the **Current Release Branch**. This raw list is for inclusion in its own section. 5. **Categorize and Rewrite for Stakeholders:** - Group the synthesized changes from Step 3 into the primary categories: `New Features`, `Enhancements`, `Bug Fixes`, and `Technical Improvements`. - Rewrite each item for the product/stakeholder audience, focusing on user benefits. **You must generate this content in both English and Japanese.** 6. **Perform Risk and Dependency Analysis for Testing Team:** - For each major impacted area identified in your synthesis, perform the following detailed analysis: - **Risk Level:** Assign a risk level: `High`, `Medium`, or `Low`. Base this on the change's complexity, its centrality to the application (e.g., changes to authentication are `High` risk), and the potential for unintended side effects. - **Components and Dependencies:** Clearly state the primary component that was changed. Then, list other key components or features that depend on it. - **You must generate this entire analysis in both English and Japanese.** 7. **Assemble the Final File:** - Format all generated bilingual content into a single Markdown file named `RELEASE_NOTES_{Identified_Version}.md`. This file is your final output. - Set the release date to today's date: **August 28, 2025**. ## 4. OUTPUT FORMAT Produce a single Markdown file named `RELEASE_NOTES_{Identified_Version}.md`. The content must strictly follow this bilingual template, with the Japanese section nested inside a collapsible `<details>` tag. # Release Notes: {Project_Name} v{Current_Version} **Generate Date:** August 28, 2025 ## Summary A brief, one-to-two-sentence overview of the most important changes included in this release version. --- ## ✨ New Features - [A clear, benefit-oriented description of the first new feature.] - [Description of the second new feature.] ## 🚀 Enhancements - [Description of an enhancement to an existing feature.] - [Description of a performance or UI improvement.] ## 🐛 Bug Fixes - [A simple explanation of the bug that was fixed, e.g., "Fixed an issue where reports would fail to generate for users in the 'Guest' role."] - [Another bug fix description.] ## 🛠️ Technical Improvements - [A high-level summary of a technical or backend change, e.g., "Upgraded the database engine to improve overall application performance and security."] - [Another technical improvement summary.] --- ## 📦 Included Pull Requests A complete list of Pull Requests included in this release. - [PR #{number}] {PR_Title} - [PR #{number}] {PR_Title} - [PR #{number}] {PR_Title} --- ## 🧪 For the Testing Team: Risk & Impact Analysis ### High-Risk Areas - **[Impacted Module/Feature 1]:** [Brief English description of the change.] - **Risk Level:** `High` - **Dependencies:** [List of dependent components, e.g., "User Profile, Reporting Module, API v2"] ### Medium-Risk Areas - **[Impacted Module/Feature 2]:** [Brief English description of the change.] - **Risk Level:** `Medium` - **Dependencies:** [List of dependent components] ### Low-Risk Areas - **[Impacted Module/Feature 3]:** [Brief English description of the change.] - **Risk Level:** `Low` - **Dependencies:** [List of dependent components] --- <details> <summary>🇯🇵 日本語のリリースノート (Japanese Release Notes)</summary> # リリースノート: {Project_Name} v{Current_Version} **作成日:** 2025年8月28日 ## 概要 (Summary) このリリースに含まれる最も重要な変更点の概要（1〜2文程度）。 --- ## ✨ 新機能 (New Features) - [新機能1の分かりやすく、メリット中心の説明。] - [新機能2の説明。] ## 🚀 機能改善 (Enhancements) - [既存機能の改善に関する説明。] - [パフォーマンスやUIの改善に関する説明。] ## 🐛 不具合修正 (Bug Fixes) - [修正された不具合の簡単な説明。例：「ゲストロールのユーザーがレポートを生成できない問題を修正しました。」] - [その他の不具合修正に関する説明。] ## 🛠️ 技術的改善 (Technical Improvements) - [技術的な変更やバックエンドの変更に関する概要。例：「システムのセキュリティを向上させるため、コアサーバーの依存関係を更新しました。」] - [その他の技術的改善に関する概要。] --- ## 📦 含まれるプルリクエスト (Included Pull Requests) このリリースに含まれるプルリクエストの完全なリストです。 - [PR #{number}] {PR_Title} - [PR #{number}] {PR_Title} - [PR #{number}] {PR_Title} --- ## 🧪 テストチーム向け: リスクと影響範囲の分析 (Risk & Impact Analysis) ### 高リスク領域 (High-Risk Areas) - **[影響を受けるモジュール/機能1]:** [変更内容の簡単な日本語説明。] - **リスクレベル (Risk Level):** `高` - **依存関係 (Dependencies):** [依存するコンポーネントのリスト] ### 中リスク領域 (Medium-Risk Areas) - **[影響を受けるモジュール/機能2]:** [変更内容の簡単な日本語説明。] - **リスクレベル (Risk Level):** `中` - **依存関係 (Dependencies):** [依存するコンポーネントのリスト] ### 低リスク領域 (Low-Risk Areas) - **[影響を受けるモジュール/機能3]:** [変更内容の簡単な日本語説明。] - **リスクレベル (Risk Level):** `低` - **依存関係 (Dependencies):** [依存するコンポーネントのリスト] </details> --- ## ACTION REQUIRED Execute the analysis and generate the bilingual release notes file as the final output. **REPOSITORY_NAME:** `{{Repository}}` **RELEASE_BRANCH_NAME:** `{{Release branch}}` ## Forbidden actions do not push to any branch
In practice, the most important part is step 3 combining intent with actual implementation. That’s what prevents misleading summaries.
Impact
QA got a reliable way to verify changes
Instead of digging through commits from scratch, QA now uses release notes as a baseline for double verification.
They can:
Cross-check listed changes against actual behavior
Focus on high-risk areas first
Understand dependencies between components
This didn’t just make things faster it made validation more structured and less error-prone.
Better visibility for stakeholders
Product managers get clear, structured release notes automatically.
- The categorization is consistent, and the language is adapted for non-engineering audiences.
Consistency across releases
Every release note now follows the same structure:
Same sections
Same formatting
Same level of detail
This makes it much easier to scan and compare releases over time.
Reduced ambiguity
We used to see a lot of questions like:
- Was this included in the release?
- Did we fix this issue?
- Is this area risky?
Now those answers are usually already in the release notes.
Lessons Learned
1. Version comparison is the foundation
The biggest improvement came from switching to version-based diffing.
Once we relied on:
- Previous tag → Current release
Everything became more accurate and complete.
2. Risk analysis adds real value
Release notes alone are useful, but adding risk analysis made them actionable.
QA can immediately identify:
- Where failures are more likely
- What areas need deeper testing
Which components are interconnected
Recommended Test Focus: Suggest whether this change requires deep functional testing, broad UI regression, or specific edge-case validation (e.g., "Focus on data integrity for Guest roles" or "Perform wide CSS regression across mobile viewports").
3. Fully automated doesn’t mean zero oversight
We still review release notes occasionally, especially for larger changes.
But instead of writing everything manually, we’re reviewing something that’s already mostly correct.
Closing Thoughts
This system isn’t complex.
It’s mostly:
- Git diffing
- A structured prompt
- A simple automation flow
But it changed how we handle releases.
Release notes went from being an unreliable afterthought to something QA can actually use for validation.
If you’re still writing release notes manually, start with version comparison. That alone solves most of the problem. The rest is just refinement.
