Self-Healing Test Automation with AI

1Timeline of a brittle selector:
2
3Day 1: Write test
4selector = "button.checkout-btn" ✓ Works
5
6Day 30: Frontend dev refactors CSS
7selector = "button.checkout-btn" ✗ Class removed, replaced with data-test-id
8
9Day 31: QA gets Slack alert: "Tests failing"
10Automation engineer: Manual hunt for new selector
11Time lost: 30 min per broken selector × 50 selectors = 25 hours/sprint
12
13Opportunity cost: Could have written new tests instead

1Test Execution:
21. Try original selector
32. Selector fails (element not found)
43. AI kicks in:
5   - Analyze current DOM
6   - Find semantically similar element
7   - Generate new selector
8   - Verify new selector works (single click test)
94. Update stored selector
105. Log repair + notify QA lead
116. Continue test execution
127. Next run: Uses repaired selector
13
14Result: Test passed (repaired), developer notified of DOM change

1# Original selector breaks
2original = "button.btn-primary:nth-child(2)"
3
4# AI tries these in order:
5strategies = [
6    "button:contains('Add to Cart')",      # Text-based
7    "button[data-testid='add-btn']",       # Test ID
8    "button[aria-label='Add to Cart']",    # Accessibility
9    "//button[.//span[text()='Add']]",     # XPath with text
10    "button.checkout-action",              # Updated class
11]
12
13# Pick first match; log which worked

1# Original: Find checkout button near total price
2
3# If broken, use context:
4def find_button_near_price(page, target_text):
5    # Find price element
6    price_elem = page.locator("text=Total:")
7    
8    # Find nearby button (parent or sibling)
9    parent = price_elem.evaluate("el => el.closest('form')")
10    button = parent.locator(f"button:has-text('{target_text}')")
11    return button

1# 1. Screenshot current page
2screenshot = page.screenshot()
3
4# 2. Ask Claude: "Where is the checkout button?"
5# 3. Claude returns coordinates or region
6
7# 4. Use AI-identified region to narrow selector search
8# 5. Scan that region for clickable elements
9
10# Reduces false positives by bounding search

1from passmark import PlaywrightTest
2
3class CheckoutTest(PlaywrightTest):
4    
5    async def test_checkout_flow(self):
6        page = self.page
7        
8        # Regular Playwright code
9        await page.goto("https://shop.example.com/checkout")
10        
11        # Passmark wraps the locator
12        # If selector fails, it auto-repairs + verifies
13        checkout_btn = self.locator("button.checkout-btn", healing=True)
14        
15        # If .checkout-btn no longer exists:
16        # 1. Passmark analyzes DOM
17        # 2. Finds button with similar role/text
18        # 3. Auto-generates new selector
19        # 4. Verifies new selector works
20        # 5. Uses new selector going forward
21        await checkout_btn.click()
22        
23        # Passmark logs the repair:
24        # {
25        #   "original": "button.checkout-btn",
26        #   "repaired": "button[data-testid='checkout']",
27        #   "confidence": 0.95,
28        #   "method": "text_match"
29        # }

1import anthropic
2from playwright.async_api import Page
3
4class SelfHealingLocator:
5    def __init__(self, page: Page, original_selector: str):
6        self.page = page
7        self.original_selector = original_selector
8        self.repair_log = []
9    
10    async def get_element(self):
11        # Try original first
12        try:
13            elements = self.page.locator(self.original_selector)
14            if await elements.count() > 0:
15                return elements
16        except:
17            pass
18        
19        # Original failed; time to heal
20        print(f"Selector broken: {self.original_selector}")
21        new_selector = await self._repair_selector()
22        
23        if new_selector:
24            self.repair_log.append({
25                "original": self.original_selector,
26                "repaired": new_selector,
27                "timestamp": datetime.now()
28            })
29            self.original_selector = new_selector  # Update for next run
30            return self.page.locator(new_selector)
31        else:
32            raise Exception(f"Could not repair selector")
33    
34    async def _repair_selector(self):
35        """Use AI to generate new selector."""
36        
37        # Get current page HTML (limited to relevant section)
38        page_html = await self.page.content()
39        
40        client = anthropic.Anthropic()
41        response = client.messages.create(
42            model="claude-opus-4-7",
43            max_tokens=500,
44            messages=[
45                {
46                    "role": "user",
47                    "content": f"""Given this HTML:
48{page_html[:2000]}
49
50The original Playwright selector '{self.original_selector}' no longer works (element not found).
51
52Generate 3 alternative CSS or XPath selectors that would find the same element:
53- Consider text content (button text, labels)
54- Consider aria attributes
55- Consider data-testid attributes
56- Consider element hierarchy
57
58Return ONLY the 3 selectors, one per line, in order of reliability:
59"""
60                }
61            ]
62        )
63        
64        candidates = response.content[0].text.strip().split('\n')
65        
66        # Verify each candidate
67        for selector in candidates:
68            try:
69                locator = self.page.locator(selector.strip())
70                count = await locator.count()
71                if count == 1:  # Found exactly one match
72                    return selector.strip()
73            except:
74                continue
75        
76        return None  # Could not repair

1Old workflow: 50 broken tests × 30 min = 25 hours/sprint
2New workflow: 50 tests auto-repair in 50 sec; engineer reviews logs = 1 hour/sprint
3Savings: 24 hours/sprint = ~6 hours/week

1class AutomationFramework:
2    def __init__(self):
3        self.repair_db = {}  # { original_selector: new_selector }
4        self.ai_client = anthropic.Anthropic()
5    
6    def locator(self, selector, page, description=""):
7        """Wraps Playwright locator with self-healing."""
8        return SelfHealingLocator(page, selector, self.ai_client, description)
9    
10    def get_repair_report(self):
11        """Generate report of auto-repairs for review."""
12        return {
13            "total_repairs": len(self.repair_db),
14            "repairs": self.repair_db,
15            "review_url": f"{self.dashboard_url}/repairs"
16        }

1# Broken selector: "div.product:nth-child(5)"
2# Product at position 5 no longer exists
3
4# Self-healing uses product name + context:
5new_selector = "div.product:has(h2:contains('Samsung TV 55in'))"
6
7# This selector is:
8# 1. More stable (based on content, not position)
9# 2. More readable (QA can understand what it's testing)
10# 3. Less fragile (works even if product order changes)

1# Selector works, but:
2# - Element was hidden (display: none)
3# - Element was disabled
4# - Element was outside viewport
5
6# Self-healing adapts:
7# Original: "button.submit"
8# Repaired: "button.submit:visible:enabled"
9
10# Reduces flakiness by making expectations explicit

1   CSS: button.btn-primary:nth-child(2)
2   XPath: //button[contains(@class, 'primary')]
3   Text: button:has-text('Add to Cart')

1   await page.click("button.checkout")

1DON'T use self-healing for:
2
31. Critical financial tests (payment, transfers)
4   - Too risky if AI picks wrong element
5   - Manual verification required
6
72. Security-sensitive tests (login, password reset)
8   - Repair might click phishing link instead of real button
9   - High stakes for error
10
113. Tests with no unique identifiers
12   - e.g., "Click any red button"
13   - Too ambiguous for AI to repair safely
14
154. Frequently changing UI
16   - Every sprint, new CSS = constant repairs
17   - Signals test design is fragile, not selector issue

1DO use self-healing for:
2
31. Utility features (settings, navigation, back buttons)
4   - Low risk if selector is slightly off
5   - High volume of tests affected by refactors
6
72. Data-driven tests (product listings, search results)
8   - Selectors often break due to pagination, filtering
9   - Self-healing adapts to new layouts
10
113. High-maintenance suites (100+ tests, frequent updates)
12   - ROI is high (saves 20+ hours/sprint)
13   - AI repair gives more attention to writing new tests
14
154. Teams with limited QA engineering bandwidth
16   - Reduce context-switching from maintenance to new features