๐ง The Engine Room: Where Logic Meets Survival
Welcome back to Part 3! In Part 1, we built the Brain (Folder Structure). In Part 2, we built the Face (Frontend & Safety Switch). Now, we build the Heartโthe Backend Logic.
Many beginners write code that looks like this:
“Loop through 500 S&P 500 stocks and fetch data for each.”
Result? You get hit with a 429 Too Many Requests error, your IP gets banned by Yahoo Finance, and your script crashes. ๐ฅ
Today, we are writing Backend_Rules.md. This file teaches your AI how to be a “Stealth Ninja”โgetting data quietly, efficiently, and without paying a cent in unnecessary API costs.
๐๏ธ The “Filter Chain” Architecture
We don’t just call APIs. We force every request to pass through a strict “Bouncer” (Security Guard). This is what we call the Filter Chain.
The 4-Step Checklist before any Network Call:
- Static Check: Is the symbol valid? (e.g., ‘AAPL’ is valid, ‘APPLE’ is not). Don’t bother the server with typos.
- Local Cache: Do I already have this data from 5 minutes ago? If yes, use it. (Cost: $0).
- Budget Check: Do I have enough “Daily Quota” left? If I’m at 99%, stop immediately.
- API Execution: Only if all above pass, we send the request.
๐ต๏ธ Anti-Ban Strategy (US Market Edition)
In the US, data providers (like SEC Edgar, Yahoo, Polygon) are smart. They know if you are a bot. If you request data at exact 1.00-second intervals, you will be blocked.
The Antigravity Solution:
- Jitter: We add random sleep times (e.g., wait 1.2s, then 3.5s, then 2.1s).
- User-Agent Rotation: We pretend to be a Chrome browser, then Firefox, then Safari.
๐ Action Item: Copy & Paste This!
Create a new file named Backend_Rules.md inside your ./agent/ folder. Paste the following rules. I have optimized them for Python, Pandas, and US Market APIs.
๐ Copy this content into your file:
# ======================================================
# ๐ Antigravity Backend Rules (US Market Edition)
# Version: 2.1
# Context: High-Availability Backend & Anti-Ban Architecture
# ======================================================
metadata:
trigger: "always_on"
description: "Distributed Systems Engineering focusing on Scraping Defense and Resource Governance for US Markets."
system_role:
title: "Principal Backend Architect"
core_objective: "Act as a shield against IP bans (SEC/Yahoo) and API Quota exhaustion."
# ------------------------------------------------------
# 1. Prime Directive: "Local First, API Last"
# ------------------------------------------------------
execution_order_strategy:
philosophy: "Strictly order logic to ensure API calls are the absolute last step."
filter_chain:
step_1:
action: "Static Validation"
details: "Check market hours (NYSE/NASDAQ), valid ticker format (e.g., $AAPL)."
cost: 0
step_2:
action: "Local Cache"
details: "Check DB/Redis/File for recent data within TTL (Time-To-Live)."
cost: 0
step_3:
action: "Budget Check"
details: "Check if 'daily_api_count' < limit."
cost: 0
step_4:
action: "Strategy Calculation"
details: "Calculate indicators using *existing* data before fetching new data."
cost: 0
step_5:
action: "EXTERNAL API CALL"
condition: "Only executed if ALL previous steps pass."
cost: "High (Network + Quota)"
# ------------------------------------------------------
# 2. Traffic Shaping (Anti-Ban & Anti-Scraping)
# ------------------------------------------------------
traffic_shaping:
context: "Sensitive libraries like 'yfinance' (Yahoo), 'ccxt' (Crypto), or SEC EDGAR scraping."
human_like_jitter:
rule: "NEVER loop with fixed intervals."
implementation: "await asyncio.sleep(base_delay + random.uniform(1.5, 4.0))"
yfinance_specific: "Minimum interval between calls > 2.0 seconds to avoid 429 errors."
headers:
user_agent_rotation: "Mandatory for HTTP requests to avoid 403 Forbidden."
circuit_breaker:
trigger: "On HTTP 403 (Forbidden) or 429 (Too Many Requests)"
action: "HALT the specific module for 1 hour."
policy: "Do not retry immediately."
# ------------------------------------------------------
# 3. Component Architecture
# ------------------------------------------------------
components:
dynamic_provider:
interface: "DataProvider"
implementations:
mock_provider:
source: "Reads JSON from './agent/mocks/'"
usage: "Used in SIMULATION mode (Cost: $0)."
real_provider:
source: "Wraps external calls (yfinance/ccxt)"
features: ["Retry (max=2)", "Structured Logging"]
quota_manager:
pattern: "Singleton"
action: "QuotaManager.check_and_consume() before every call."
failure_mode: "If limit exceeded, raise 'QuotaExceededException' and fallback to cached data."
# ------------------------------------------------------
# โก Special Commands
# ------------------------------------------------------
commands:
architect_mode:
action: "Analyze current code and refactor to move API calls inside the deepest execution block."๐ง Why This File is Powerful
By saving this, you have programmed your AI to:
- Save Money: It will look for local files before spending your API budget.
- Stay Hidden: It will act like a human (random delays) to avoid IP bans.
- Be Safe: It separates “Testing” (Mock) from “Real Money” (RealProvider).
๐ The Final Step
We have the Brain, the Face, and the Heart. But how do they talk to each other? How do we actually build features without breaking things?
In the final Part 4, we will define the Workflow_Rules.md. This is the “Team Handbook” that tells the AI how to behave, how to write tests, and how to verify its own code.
Your bot is almost alive. See you in the finale! ๐