diff --git a/skills/dcf-valuation/SKILL.md b/skills/dcf-valuation/SKILL.md new file mode 100644 index 00000000..31f9ff50 --- /dev/null +++ b/skills/dcf-valuation/SKILL.md @@ -0,0 +1,213 @@ +--- +name: DCF Valuation +description: Perform Discounted Cash Flow (DCF) valuation analysis for public companies. Use when the user asks to value a stock, calculate intrinsic value, fair value, perform DCF analysis, determine if a stock is undervalued or overvalued, or estimate a price target. +version: 1.1.1 +metadata: + emoji: "\U0001F9EE" + tags: + - finance + - valuation + - dcf +userInvocable: true +disableModelInvocation: false +--- + +## Instructions + +Perform a rigorous Discounted Cash Flow (DCF) valuation. Follow all steps and show your work. Use external macro context when assumptions are time-sensitive (for example, risk-free rate regime shifts). + +### Progress Checklist + +``` +DCF Analysis Progress: +- [ ] Step 1: Gather financial data +- [ ] Step 2: Calculate historical FCF and growth +- [ ] Step 3: Estimate WACC +- [ ] Step 4: Project future cash flows +- [ ] Step 5: Calculate present value and fair value +- [ ] Step 6: Sensitivity analysis +- [ ] Step 7: Validate results +- [ ] Step 8: Present findings +``` + +### Step 1: Gather Financial Data + +Use `data` tool with `domain="finance"` for all calls: + +1. **Cash Flow History** (5 years): + ``` + action: "get_cash_flow_statements" + params: { ticker: "[TICKER]", period: "annual", limit: 5 } + ``` + Extract: `free_cash_flow`, `net_cash_flow_from_operations`, `capital_expenditure` + Fallback: FCF = Operating Cash Flow - CapEx + +2. **Income Statements** (5 years): + ``` + action: "get_income_statements" + params: { ticker: "[TICKER]", period: "annual", limit: 5 } + ``` + Extract: `revenue`, `operating_income`, `net_income`, `income_tax_expense` + +3. **Balance Sheet** (latest): + ``` + action: "get_balance_sheets" + params: { ticker: "[TICKER]", period: "annual", limit: 1 } + ``` + Extract: `total_debt`, `cash_and_equivalents`, `outstanding_shares` + +4. **Financial Metrics** (current): + ``` + action: "get_financial_metrics_snapshot" + params: { ticker: "[TICKER]" } + ``` + Extract: `market_cap`, `enterprise_value`, `return_on_invested_capital`, `debt_to_equity`, `free_cash_flow_per_share` + +5. **Analyst Estimates**: + ``` + action: "get_analyst_estimates" + params: { ticker: "[TICKER]", period: "annual" } + ``` + Extract: Forward EPS estimates for growth validation + +6. **Current Price**: + ``` + action: "get_price_snapshot" + params: { ticker: "[TICKER]" } + ``` + +7. **Company Facts**: + ``` + action: "get_company_facts" + params: { ticker: "[TICKER]" } + ``` + Extract: `sector` β€” use to determine WACC range from [sector-wacc.md](references/sector-wacc.md) + +8. **Recent Event Context**: +- Pull company-specific headlines with: + ``` + action: "get_news" + params: { ticker: "[TICKER]", limit: 10 } + ``` +- Use this to flag event risk (guidance reset, litigation, regulation, one-off gains/losses) that may distort near-term FCF extrapolation. + +### Step 2: Calculate Historical FCF and Growth + +- Compute FCF for each of the last 5 years +- Calculate 5-year FCF CAGR: `(FCF_latest / FCF_earliest)^(1/years) - 1` +- Cross-validate with: revenue growth, operating income growth, analyst EPS growth +- **Cap projected growth at 15%** (sustained higher growth is rare) +- If FCF is volatile, weight analyst estimates more heavily + +### Step 3: Estimate WACC + +Use the company's `sector` to look up the base WACC range from [sector-wacc.md](references/sector-wacc.md). + +**Calculate WACC:** +``` +WACC = (E/V) * Re + (D/V) * Rd * (1 - Tax Rate) + +Where: + E = Market cap (equity value) + D = Total debt + V = E + D + Re = Risk-free rate + Beta * Equity Risk Premium + Rd = Cost of debt (estimate from interest expense / total debt) + Tax Rate = Effective tax rate from income statements +``` + +**Default assumptions:** +- Risk-free rate: pull latest 10-year Treasury yield using `web_search` (preferred) and cite date/source. Fallback range: ~4.0-4.5%. +- Equity risk premium: ~5.5% +- If beta unavailable, use sector average + +**Sanity check:** WACC should be 2-4% below ROIC for value-creating companies. + +### Step 4: Project Future Cash Flows (Years 1-5) + +- Apply growth rate with annual decay (multiply by 0.95 each year) +- Year 1: FCF * (1 + growth_rate) +- Year 2: FCF * (1 + growth_rate * 0.95) +- Year 3: FCF * (1 + growth_rate * 0.90) +- Year 4: FCF * (1 + growth_rate * 0.85) +- Year 5: FCF * (1 + growth_rate * 0.80) + +**Terminal Value** (Gordon Growth Model): +``` +TV = FCF_Year5 * (1 + g) / (WACC - g) +Where g = terminal growth rate (2.5% default, GDP proxy) +``` + +### Step 5: Calculate Present Value and Fair Value + +``` +PV of each FCF = FCF_t / (1 + WACC)^t +PV of Terminal Value = TV / (1 + WACC)^5 + +Enterprise Value = Sum of PV(FCFs) + PV(Terminal Value) +Net Debt = Total Debt - Cash and Equivalents +Equity Value = Enterprise Value - Net Debt +Fair Value per Share = Equity Value / Shares Outstanding +``` + +### Step 6: Sensitivity Analysis + +Create a matrix varying two key assumptions: + +| | TG 2.0% | TG 2.5% | TG 3.0% | +|---|---|---|---| +| **WACC -1%** | $ | $ | $ | +| **WACC base** | $ | $ | $ | +| **WACC +1%** | $ | $ | $ | + +(TG = Terminal Growth Rate) + +### Step 7: Validate Results + +Before presenting, check: + +1. **EV comparison**: Calculated EV within 30% of reported enterprise_value + - If off by >30%, revisit WACC or growth assumptions +2. **Terminal value ratio**: Should be 50-80% of total EV for mature companies + - If >90%, growth rate may be too high + - If <40%, near-term projections may be aggressive +3. **FCF yield check**: Compare fair value FCF yield to current market FCF yield + +If validation fails, adjust assumptions and recalculate. + +### Step 8: Present Results + +Format clearly with: + +1. **Executive Summary** + - Current price vs. fair value estimate + - Upside/downside percentage + - Verdict: Undervalued / Fairly Valued / Overvalued + +2. **Key Assumptions Table** + | Assumption | Value | Source | + |---|---|---| + | Growth Rate | X% | 5Y CAGR + analyst cross-check | + | WACC | X% | Sector range + company adjustments | + | Terminal Growth | X% | GDP proxy | + | Tax Rate | X% | Effective rate from financials | + +3. **Projected FCF Table** + | Year | FCF | Growth | PV of FCF | + |---|---|---|---| + +4. **Valuation Bridge** + - PV of projected FCFs + - PV of Terminal Value + - = Enterprise Value + - - Net Debt + - = Equity Value + - / Shares Outstanding + - = **Fair Value per Share** + +5. **Sensitivity Matrix** (from Step 6) + +6. **Risks & Caveats** + - Key risks to the valuation thesis + - DCF limitations (sensitive to growth and WACC assumptions) + - Company-specific caveats (high debt, cyclicality, early-stage, etc.) diff --git a/skills/dcf-valuation/references/sector-wacc.md b/skills/dcf-valuation/references/sector-wacc.md new file mode 100644 index 00000000..5ddc1bd1 --- /dev/null +++ b/skills/dcf-valuation/references/sector-wacc.md @@ -0,0 +1,40 @@ +# Sector WACC Reference + +Use the company's `sector` from `get_company_facts` to look up the base WACC range below, then adjust for company-specific factors. + +## WACC by Sector + +| Sector | Typical WACC Range | Notes | +|--------|-------------------|-------| +| Communication Services | 8-10% | Mix of stable telecom and growth media | +| Consumer Discretionary | 8-10% | Cyclical exposure | +| Consumer Staples | 7-8% | Defensive, stable demand | +| Energy | 9-11% | Commodity price exposure | +| Financials | 8-10% | Leverage already in business model | +| Health Care | 8-10% | Regulatory and pipeline risk | +| Industrials | 8-9% | Moderate cyclicality | +| Information Technology | 8-12% | Higher end for high-growth; lower for mature | +| Materials | 8-10% | Cyclical, commodity exposure | +| Real Estate | 7-9% | Interest rate sensitivity | +| Utilities | 6-7% | Regulated, stable cash flows | + +## Adjustment Factors + +**Add to base WACC:** +- High debt (D/E > 1.5): +1-2% +- Small cap (< $2B market cap): +1-2% +- Emerging markets exposure: +1-3% +- Concentrated customer base: +0.5-1% +- Regulatory uncertainty: +0.5-1.5% + +**Subtract from base WACC:** +- Market leader with moat: -0.5-1% +- Recurring revenue model: -0.5-1% +- Investment grade credit: -0.5% + +## Sanity Checks + +- WACC should typically be 2-4% below ROIC for value-creating companies +- If WACC > ROIC, the company may be destroying value +- Typical range for US large-cap: 7-12% +- Anything below 6% or above 14% warrants extra scrutiny diff --git a/skills/docx/SKILL.md b/skills/docx/SKILL.md new file mode 100644 index 00000000..ce1f04e5 --- /dev/null +++ b/skills/docx/SKILL.md @@ -0,0 +1,513 @@ +--- +name: Word Document +description: "Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files). Triggers include: any mention of \"Word doc\", \"word document\", \".docx\", or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a \"report\", \"memo\", \"letter\", \"template\", or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation." +version: 1.0.0 +metadata: + emoji: "πŸ“„" + tags: + - office + - document + - docx + install: + - id: brew-pandoc + kind: brew + formula: pandoc + bins: [pandoc] + label: "Install pandoc for text extraction" + os: [darwin, linux] + - id: brew-libreoffice + kind: brew + formula: libreoffice + bins: [soffice] + label: "Install LibreOffice for PDF conversion" + os: [darwin] + - id: brew-poppler + kind: brew + formula: poppler + bins: [pdftoppm] + label: "Install poppler for PDF to image conversion" + os: [darwin, linux] + - id: npm-docx + kind: node + formula: docx + bins: [] + label: "Install docx-js for document creation" +userInvocable: true +disableModelInvocation: false +--- + +# DOCX creation, editing, and analysis + +## Overview + +A .docx file is a ZIP archive containing XML files. + +## Quick Reference + +| Task | Approach | +|------|----------| +| Read/analyze content | `pandoc` or unpack for raw XML | +| Create new document | Use `docx-js` - see Creating New Documents below | +| Edit existing document | Unpack β†’ edit XML β†’ repack - see Editing Existing Documents below | + +### Converting .doc to .docx + +Legacy `.doc` files must be converted before editing: + +```bash +python scripts/office/soffice.py --headless --convert-to docx document.doc +``` + +### Reading Content + +```bash +# Text extraction with tracked changes +pandoc --track-changes=all document.docx -o output.md + +# Raw XML access +python scripts/office/unpack.py document.docx unpacked/ +``` + +### Converting to Images + +```bash +python scripts/office/soffice.py --headless --convert-to pdf document.docx +pdftoppm -jpeg -r 150 document.pdf page +``` + +### Accepting Tracked Changes + +To produce a clean document with all tracked changes accepted (requires LibreOffice): + +```bash +python scripts/accept_changes.py input.docx output.docx +``` + +--- + +## Creating New Documents + +Generate .docx files with JavaScript, then validate. Install: `npm install -g docx` + +### Setup +```javascript +const { Document, Packer, Paragraph, TextRun, Table, TableRow, TableCell, ImageRun, + Header, Footer, AlignmentType, PageOrientation, LevelFormat, ExternalHyperlink, + TableOfContents, HeadingLevel, BorderStyle, WidthType, ShadingType, + VerticalAlign, PageNumber, PageBreak } = require('docx'); + +const doc = new Document({ sections: [{ children: [/* content */] }] }); +Packer.toBuffer(doc).then(buffer => fs.writeFileSync("doc.docx", buffer)); +``` + +### Validation +After creating the file, validate it. If validation fails, unpack, fix the XML, and repack. +```bash +python scripts/office/validate.py doc.docx +``` + +### Page Size + +```javascript +// CRITICAL: docx-js defaults to A4, not US Letter +// Always set page size explicitly for consistent results +sections: [{ + properties: { + page: { + size: { + width: 12240, // 8.5 inches in DXA + height: 15840 // 11 inches in DXA + }, + margin: { top: 1440, right: 1440, bottom: 1440, left: 1440 } // 1 inch margins + } + }, + children: [/* content */] +}] +``` + +**Common page sizes (DXA units, 1440 DXA = 1 inch):** + +| Paper | Width | Height | Content Width (1" margins) | +|-------|-------|--------|---------------------------| +| US Letter | 12,240 | 15,840 | 9,360 | +| A4 (default) | 11,906 | 16,838 | 9,026 | + +**Landscape orientation:** docx-js swaps width/height internally, so pass portrait dimensions and let it handle the swap: +```javascript +size: { + width: 12240, // Pass SHORT edge as width + height: 15840, // Pass LONG edge as height + orientation: PageOrientation.LANDSCAPE // docx-js swaps them in the XML +}, +// Content width = 15840 - left margin - right margin (uses the long edge) +``` + +### Styles (Override Built-in Headings) + +Use Arial as the default font (universally supported). Keep titles black for readability. + +```javascript +const doc = new Document({ + styles: { + default: { document: { run: { font: "Arial", size: 24 } } }, // 12pt default + paragraphStyles: [ + // IMPORTANT: Use exact IDs to override built-in styles + { id: "Heading1", name: "Heading 1", basedOn: "Normal", next: "Normal", quickFormat: true, + run: { size: 32, bold: true, font: "Arial" }, + paragraph: { spacing: { before: 240, after: 240 }, outlineLevel: 0 } }, // outlineLevel required for TOC + { id: "Heading2", name: "Heading 2", basedOn: "Normal", next: "Normal", quickFormat: true, + run: { size: 28, bold: true, font: "Arial" }, + paragraph: { spacing: { before: 180, after: 180 }, outlineLevel: 1 } }, + ] + }, + sections: [{ + children: [ + new Paragraph({ heading: HeadingLevel.HEADING_1, children: [new TextRun("Title")] }), + ] + }] +}); +``` + +### Lists (NEVER use unicode bullets) + +```javascript +// WRONG - never manually insert bullet characters +new Paragraph({ children: [new TextRun("Item")] }) // BAD +new Paragraph({ children: [new TextRun("\u2022 Item")] }) // BAD + +// CORRECT - use numbering config with LevelFormat.BULLET +const doc = new Document({ + numbering: { + config: [ + { reference: "bullets", + levels: [{ level: 0, format: LevelFormat.BULLET, text: "\u2022", alignment: AlignmentType.LEFT, + style: { paragraph: { indent: { left: 720, hanging: 360 } } } }] }, + { reference: "numbers", + levels: [{ level: 0, format: LevelFormat.DECIMAL, text: "%1.", alignment: AlignmentType.LEFT, + style: { paragraph: { indent: { left: 720, hanging: 360 } } } }] }, + ] + }, + sections: [{ + children: [ + new Paragraph({ numbering: { reference: "bullets", level: 0 }, + children: [new TextRun("Bullet item")] }), + new Paragraph({ numbering: { reference: "numbers", level: 0 }, + children: [new TextRun("Numbered item")] }), + ] + }] +}); + +// Each reference creates INDEPENDENT numbering +// Same reference = continues (1,2,3 then 4,5,6) +// Different reference = restarts (1,2,3 then 1,2,3) +``` + +### Tables + +**CRITICAL: Tables need dual widths** - set both `columnWidths` on the table AND `width` on each cell. Without both, tables render incorrectly on some platforms. + +```javascript +// CRITICAL: Always set table width for consistent rendering +// CRITICAL: Use ShadingType.CLEAR (not SOLID) to prevent black backgrounds +const border = { style: BorderStyle.SINGLE, size: 1, color: "CCCCCC" }; +const borders = { top: border, bottom: border, left: border, right: border }; + +new Table({ + width: { size: 9360, type: WidthType.DXA }, // Always use DXA (percentages break in Google Docs) + columnWidths: [4680, 4680], // Must sum to table width (DXA: 1440 = 1 inch) + rows: [ + new TableRow({ + children: [ + new TableCell({ + borders, + width: { size: 4680, type: WidthType.DXA }, // Also set on each cell + shading: { fill: "D5E8F0", type: ShadingType.CLEAR }, // CLEAR not SOLID + margins: { top: 80, bottom: 80, left: 120, right: 120 }, // Cell padding (internal, not added to width) + children: [new Paragraph({ children: [new TextRun("Cell")] })] + }) + ] + }) + ] +}) +``` + +**Table width calculation:** + +Always use `WidthType.DXA` β€” `WidthType.PERCENTAGE` breaks in Google Docs. + +```javascript +// Table width = sum of columnWidths = content width +// US Letter with 1" margins: 12240 - 2880 = 9360 DXA +width: { size: 9360, type: WidthType.DXA }, +columnWidths: [7000, 2360] // Must sum to table width +``` + +**Width rules:** +- **Always use `WidthType.DXA`** β€” never `WidthType.PERCENTAGE` (incompatible with Google Docs) +- Table width must equal the sum of `columnWidths` +- Cell `width` must match corresponding `columnWidth` +- Cell `margins` are internal padding - they reduce content area, not add to cell width +- For full-width tables: use content width (page width minus left and right margins) + +### Images + +```javascript +// CRITICAL: type parameter is REQUIRED +new Paragraph({ + children: [new ImageRun({ + type: "png", // Required: png, jpg, jpeg, gif, bmp, svg + data: fs.readFileSync("image.png"), + transformation: { width: 200, height: 150 }, + altText: { title: "Title", description: "Desc", name: "Name" } // All three required + })] +}) +``` + +### Page Breaks + +```javascript +// CRITICAL: PageBreak must be inside a Paragraph +new Paragraph({ children: [new PageBreak()] }) + +// Or use pageBreakBefore +new Paragraph({ pageBreakBefore: true, children: [new TextRun("New page")] }) +``` + +### Table of Contents + +```javascript +// CRITICAL: Headings must use HeadingLevel ONLY - no custom styles +new TableOfContents("Table of Contents", { hyperlink: true, headingStyleRange: "1-3" }) +``` + +### Headers/Footers + +```javascript +sections: [{ + properties: { + page: { margin: { top: 1440, right: 1440, bottom: 1440, left: 1440 } } // 1440 = 1 inch + }, + headers: { + default: new Header({ children: [new Paragraph({ children: [new TextRun("Header")] })] }) + }, + footers: { + default: new Footer({ children: [new Paragraph({ + children: [new TextRun("Page "), new TextRun({ children: [PageNumber.CURRENT] })] + })] }) + }, + children: [/* content */] +}] +``` + +### Critical Rules for docx-js + +- **Set page size explicitly** - docx-js defaults to A4; use US Letter (12240 x 15840 DXA) for US documents +- **Landscape: pass portrait dimensions** - docx-js swaps width/height internally; pass short edge as `width`, long edge as `height`, and set `orientation: PageOrientation.LANDSCAPE` +- **Never use `\n`** - use separate Paragraph elements +- **Never use unicode bullets** - use `LevelFormat.BULLET` with numbering config +- **PageBreak must be in Paragraph** - standalone creates invalid XML +- **ImageRun requires `type`** - always specify png/jpg/etc +- **Always set table `width` with DXA** - never use `WidthType.PERCENTAGE` (breaks in Google Docs) +- **Tables need dual widths** - `columnWidths` array AND cell `width`, both must match +- **Table width = sum of columnWidths** - for DXA, ensure they add up exactly +- **Always add cell margins** - use `margins: { top: 80, bottom: 80, left: 120, right: 120 }` for readable padding +- **Use `ShadingType.CLEAR`** - never SOLID for table shading +- **TOC requires HeadingLevel only** - no custom styles on heading paragraphs +- **Override built-in styles** - use exact IDs: "Heading1", "Heading2", etc. +- **Include `outlineLevel`** - required for TOC (0 for H1, 1 for H2, etc.) + +--- + +## Editing Existing Documents + +**Follow all 3 steps in order.** + +### Step 1: Unpack +```bash +python scripts/office/unpack.py document.docx unpacked/ +``` +Extracts XML, pretty-prints, merges adjacent runs, and converts smart quotes to XML entities (`“` etc.) so they survive editing. Use `--merge-runs false` to skip run merging. + +### Step 2: Edit XML + +Edit files in `unpacked/word/`. See XML Reference below for patterns. + +**Use "Claude" as the author** for tracked changes and comments, unless the user explicitly requests use of a different name. + +**Use the Edit tool directly for string replacement. Do not write Python scripts.** Scripts introduce unnecessary complexity. The Edit tool shows exactly what is being replaced. + +**CRITICAL: Use smart quotes for new content.** When adding text with apostrophes or quotes, use XML entities to produce smart quotes: +```xml + +Here’s a quote: “Hello” +``` +| Entity | Character | +|--------|-----------| +| `‘` | ' (left single) | +| `’` | ' (right single / apostrophe) | +| `“` | " (left double) | +| `”` | " (right double) | + +**Adding comments:** Use `comment.py` to handle boilerplate across multiple XML files (text must be pre-escaped XML): +```bash +python scripts/comment.py unpacked/ 0 "Comment text with & and ’" +python scripts/comment.py unpacked/ 1 "Reply text" --parent 0 # reply to comment 0 +python scripts/comment.py unpacked/ 0 "Text" --author "Custom Author" # custom author name +``` +Then add markers to document.xml (see Comments in XML Reference). + +### Step 3: Pack +```bash +python scripts/office/pack.py unpacked/ output.docx --original document.docx +``` +Validates with auto-repair, condenses XML, and creates DOCX. Use `--validate false` to skip. + +**Auto-repair will fix:** +- `durableId` >= 0x7FFFFFFF (regenerates valid ID) +- Missing `xml:space="preserve"` on `` with whitespace + +**Auto-repair won't fix:** +- Malformed XML, invalid element nesting, missing relationships, schema violations + +### Common Pitfalls + +- **Replace entire `` elements**: When adding tracked changes, replace the whole `...` block with `......` as siblings. Don't inject tracked change tags inside a run. +- **Preserve `` formatting**: Copy the original run's `` block into your tracked change runs to maintain bold, font size, etc. + +--- + +## XML Reference + +### Schema Compliance + +- **Element order in ``**: ``, ``, ``, ``, ``, `` last +- **Whitespace**: Add `xml:space="preserve"` to `` with leading/trailing spaces +- **RSIDs**: Must be 8-digit hex (e.g., `00AB1234`) + +### Tracked Changes + +**Insertion:** +```xml + + inserted text + +``` + +**Deletion:** +```xml + + deleted text + +``` + +**Inside ``**: Use `` instead of ``, and `` instead of ``. + +**Minimal edits** - only mark what changes: +```xml + +The term is + + 30 + + + 60 + + days. +``` + +**Deleting entire paragraphs/list items** - when removing ALL content from a paragraph, also mark the paragraph mark as deleted so it merges with the next paragraph. Add `` inside ``: +```xml + + + ... + + + + + + Entire paragraph content being deleted... + + +``` +Without the `` in ``, accepting changes leaves an empty paragraph/list item. + +**Rejecting another author's insertion** - nest deletion inside their insertion: +```xml + + + their inserted text + + +``` + +**Restoring another author's deletion** - add insertion after (don't modify their deletion): +```xml + + deleted text + + + deleted text + +``` + +### Comments + +After running `comment.py` (see Step 2), add markers to document.xml. For replies, use `--parent` flag and nest markers inside the parent's. + +**CRITICAL: `` and `` are siblings of ``, never inside ``.** + +```xml + + + + deleted + + more text + + + + + + + text + + + + +``` + +### Images + +1. Add image file to `word/media/` +2. Add relationship to `word/_rels/document.xml.rels`: +```xml + +``` +3. Add content type to `[Content_Types].xml`: +```xml + +``` +4. Reference in document.xml: +```xml + + + + + + + + + + + + +``` + +--- + +## Dependencies + +- **pandoc**: Text extraction +- **docx**: `npm install -g docx` (new documents) +- **LibreOffice**: PDF conversion (auto-configured for sandboxed environments via `scripts/office/soffice.py`) +- **Poppler**: `pdftoppm` for images diff --git a/skills/earnings-analysis/SKILL.md b/skills/earnings-analysis/SKILL.md new file mode 100644 index 00000000..450e1f06 --- /dev/null +++ b/skills/earnings-analysis/SKILL.md @@ -0,0 +1,463 @@ +--- +name: Earnings Analysis +description: >- + Analyze a company's financial statements (income statement, balance sheet, + cash flow statement) to assess financial health, earnings quality, and + competitive advantage. Use when the user asks to read/analyze financial + statements, check earnings quality, assess financial health, evaluate + profitability trends, or screen for competitive moats. +version: 1.0.0 +metadata: + emoji: "\U0001F4D1" + requires: + env: + - FINANCIAL_DATASETS_API_KEY + tags: + - finance + - earnings + - analysis + - statements + - buffett +userInvocable: true +disableModelInvocation: false +--- + +## Instructions + +You are performing a structured financial statement analysis. Follow all steps in order and show your work. Output language must match the user's input language. + +**IMPORTANT: This analysis requires BOTH structured data AND external context.** You MUST use `web_search` to gather earnings call insights, industry context, and explanations for data anomalies. An analysis based only on API data without any web research is incomplete. Expect to make 3-6 web searches throughout the analysis. + +### Progress Checklist + +``` +Earnings Analysis Progress: +- [ ] Step 1: Gather financial data +- [ ] Step 2: Income statement analysis +- [ ] Step 3: Balance sheet analysis +- [ ] Step 4: Cash flow statement analysis +- [ ] Step 5: Buffett competitive advantage scoring +- [ ] Step 6: Quality of earnings assessment +- [ ] Step 7: SEC filing qualitative analysis +- [ ] Step 8: Peer comparison (if requested) +- [ ] Step 9: Present findings +``` + +### Step 1: Gather Financial Data + +Use `data` tool with `domain="finance"` for all structured data calls. + +#### 1a. Structured Data + +1. **Annual financial statements** (5 years): + ``` + action: "get_all_financial_statements" + params: { ticker: "[TICKER]", period: "annual", limit: 5 } + ``` + This returns income statements, balance sheets, and cash flow statements together. + +2. **Quarterly financial statements** (last 4 quarters): + ``` + action: "get_all_financial_statements" + params: { ticker: "[TICKER]", period: "quarterly", limit: 4 } + ``` + +3. **Current financial metrics**: + ``` + action: "get_financial_metrics_snapshot" + params: { ticker: "[TICKER]" } + ``` + +4. **Company facts**: + ``` + action: "get_company_facts" + params: { ticker: "[TICKER]" } + ``` + Extract: `sector`, `industry` β€” needed for benchmark comparisons in later steps. + +5. **Current stock price**: + ``` + action: "get_price_snapshot" + params: { ticker: "[TICKER]" } + ``` + +6. **Recent news**: + ``` + action: "get_news" + params: { ticker: "[TICKER]", limit: 10 } + ``` + Scan headlines for material events (earnings surprises, guidance changes, M&A, restructuring). + +#### 1b. External Context (Web Search) β€” MANDATORY + +You MUST run the following two web searches after gathering structured data. These are not optional. + +1. **Latest earnings call highlights** (REQUIRED): + ``` + web_search("[COMPANY] latest earnings call highlights key takeaways [CURRENT_YEAR]") + ``` + Extract: management guidance, segment commentary, strategic priorities, forward outlook. + This provides the "why" behind the numbers that structured data cannot explain. + +2. **Industry/macro backdrop** (REQUIRED): + ``` + web_search("[INDUSTRY] industry outlook trends [CURRENT_YEAR]") + ``` + Extract: industry growth rate, tailwinds/headwinds, regulatory changes, competitive dynamics. + This is needed to assess whether the company's performance is company-specific or industry-wide. + +3. **Company-specific events** (conditional β€” run if news headlines or data show a material event): + ``` + web_search("[COMPANY] [EVENT_KEYWORD] impact analysis") + ``` + Examples: acquisition, restructuring, product launch, lawsuit, management change. + +**Checkpoint:** Before proceeding to Step 2, verify that you have completed at least 2 web searches above. If you have not, go back and run them now. + +### Step 2: Income Statement Analysis + +Analyze the income statement across all 5 annual periods. Calculate and present: + +1. **Revenue trend**: + - Year-over-year growth rate for each year + - 5-year CAGR: `(Revenue_latest / Revenue_earliest)^(1/years) - 1` + - Flag any years with revenue decline + +2. **Margin analysis** (calculate for each year, show the trend): + - Gross Margin = Gross Profit / Revenue + - Operating Margin = Operating Income / Revenue + - Net Margin = Net Income / Revenue + +3. **Margin benchmarks** (from [financial-ratios-benchmarks.md](references/financial-ratios-benchmarks.md)): + - Compare each margin to sector benchmarks + - Flag margins that are significantly above or below sector range + +4. **EPS analysis**: + - EPS trend over 5 years + - EPS growth consistency (note any years of decline) + +5. **Expense structure**: + - Cost of revenue as % of revenue (trend) + - SG&A as % of revenue (trend) + - R&D as % of revenue (trend, if applicable) + - Flag any expense category growing faster than revenue + +6. **Contextual explanation** (REQUIRED β€” use web search results from Step 1b): + - For each significant trend or inflection point in the data above, provide a **why** explanation using the earnings call and industry context gathered in Step 1b. + - If revenue growth changed direction significantly (acceleration or deceleration > 10pp), run an additional search: + `web_search("[COMPANY] revenue [growth/decline] reason [YEAR]")` + - If margins shifted by more than 5pp year-over-year, run an additional search: + `web_search("[COMPANY] margin [expansion/compression] [YEAR]")` + - **Do not present a data table without narrative.** Every major trend must have a "why" attached, citing the source (earnings call, industry report, or company announcement). + +Present as a table: + +| Metric | Year 1 | Year 2 | Year 3 | Year 4 | Year 5 | 5Y CAGR | +|--------|--------|--------|--------|--------|--------|---------| + +### Step 3: Balance Sheet Analysis + +Analyze the balance sheet across all 5 annual periods: + +1. **Liquidity**: + - Current Ratio = Current Assets / Current Liabilities + - Quick Ratio = (Current Assets - Inventory) / Current Liabilities + - Cash and equivalents trend + +2. **Leverage**: + - Cash vs. Total Debt (short-term + long-term debt) + - Debt-to-Equity = Total Liabilities / Total Shareholders' Equity + - Interest Coverage = Operating Income / Interest Expense + - Debt payoff capacity = Total Debt / Net Income (in years) + +3. **Asset quality**: + - Receivables Turnover = Revenue / Accounts Receivable + - Inventory Turnover = Cost of Revenue / Inventory (if applicable) + - Goodwill as % of Total Assets (flag if > 30%) + +4. **Equity structure**: + - Retained earnings: year-over-year changes (growing?) + - Preferred stock: present or absent? + - Treasury stock: present? growing? (indicates buybacks) + +5. **Working capital trend**: + - Net Working Capital = Current Assets - Current Liabilities + - Direction of change over 5 years + +6. **Contextual explanation** (use web search results from Step 1b + additional searches as needed): + - Explain major balance sheet changes using earnings call context from Step 1b. + - If total debt changed significantly (> 30% YoY), you MUST search for the reason: + `web_search("[COMPANY] debt [issuance/repayment] [YEAR]")` + - If goodwill jumped, you MUST search for acquisition context: + `web_search("[COMPANY] acquisition [YEAR]")` + - Large treasury stock changes β†’ confirm buyback program details: + `web_search("[COMPANY] share buyback program")` + +Compare key ratios to sector benchmarks from [financial-ratios-benchmarks.md](references/financial-ratios-benchmarks.md). + +### Step 4: Cash Flow Statement Analysis + +Analyze cash flow statements across all 5 annual periods: + +1. **Operating cash flow quality**: + - OCF vs. Net Income ratio for each year + - Target: OCF/NI > 1.0 (cash earnings exceed accrual earnings) + - Trend direction + +2. **Free cash flow**: + - FCF = Operating Cash Flow - Capital Expenditure + - FCF Margin = FCF / Revenue + - 5-year FCF trend and CAGR + +3. **Capital intensity**: + - CapEx / Revenue ratio + - CapEx / Net Income ratio (Buffett benchmark: < 25% excellent, < 50% acceptable) + - Is CapEx growing faster than revenue? (potential red flag) + +4. **Cash flow composition**: + - Net cash from operating activities (should be consistently positive) + - Net cash from investing activities (negative = investing in growth) + - Net cash from financing activities (pattern: debt vs. equity funded?) + +5. **Shareholder returns**: + - Dividends paid (from financing activities) + - Share buybacks / treasury stock repurchase + - Total payout ratio = (Dividends + Buybacks) / Net Income + - Is the company returning cash while maintaining growth? + +6. **Contextual explanation** (use web search results from Step 1b + additional searches as needed): + - Explain cash flow patterns using earnings call context from Step 1b. + - If CapEx spiked significantly in a particular year, you MUST search for what was built: + `web_search("[COMPANY] capital expenditure investment [YEAR]")` + - If FCF diverged sharply from net income, search for restructuring or working capital events. + +Present a summary table: + +| Metric | Year 1 | Year 2 | Year 3 | Year 4 | Year 5 | +|--------|--------|--------|--------|--------|--------| + +### Step 5: Buffett Competitive Advantage Scoring + +Apply the scoring framework from [buffett-checklist.md](references/buffett-checklist.md). + +For each of the 13 criteria across 4 categories: +1. Calculate the metric value from the data gathered in Steps 1-4 +2. Determine the score based on the threshold table +3. Note the sector-specific caveats (Financials, Utilities, REITs, Growth-stage) + +Present the full scorecard table and the overall rating (Excellent / Good / Average / Weak). + +### Step 6: Quality of Earnings Assessment + +Assess whether reported earnings are backed by real cash and sustainable operations: + +1. **Accrual ratio**: + - Formula: (Net Income - Operating Cash Flow) / Total Assets + - Interpretation: Lower is better. High positive values suggest earnings are driven by accruals rather than cash. + - Red flag threshold: > 10% + +2. **Revenue recognition quality**: + - Compare Accounts Receivable growth rate vs. Revenue growth rate + - If AR grows significantly faster than revenue β†’ potential aggressive revenue recognition + - Red flag threshold: AR growth > Revenue growth + 5 percentage points + +3. **Inventory quality** (if applicable): + - Compare Inventory growth rate vs. Cost of Revenue growth rate + - Rising inventory vs. flat/declining COGS β†’ potential obsolescence risk + - Red flag threshold: Inventory growth > COGS growth + 10 percentage points + +4. **One-time items**: + - Identify significant non-recurring charges or gains in the income statement + - Calculate adjusted net income excluding one-time items + - Compare adjusted vs. reported margins + +5. **Deferred revenue trend** (if applicable): + - Growing deferred revenue is a positive signal (future revenue already contracted) + - Declining deferred revenue may signal weakening demand pipeline + +6. **External validation** (web search): + - If any red flags were triggered above, search for corroborating or mitigating context: + `web_search("[COMPANY] accounting concerns OR restatement OR SEC inquiry")` + - Check for auditor changes (can signal accounting issues): + `web_search("[COMPANY] auditor change OR audit opinion")` + - Only run these searches if quantitative red flags exist. Do not search proactively for every company. + +Summarize quality of earnings as: **High** / **Moderate** / **Low** with supporting evidence. + +### Step 7: SEC Filing Qualitative Analysis + +Pull and analyze the most recent annual or quarterly filing: + +1. **Get filing list**: + ``` + action: "get_filings" + params: { ticker: "[TICKER]", filing_type: "10-K", limit: 1 } + ``` + If 10-K is not recent enough, also pull 10-Q: + ``` + action: "get_filings" + params: { ticker: "[TICKER]", filing_type: "10-Q", limit: 1 } + ``` + +2. **Read MD&A section** (Management's Discussion and Analysis): + ``` + action: "get_filing_items" + params: { ticker: "[TICKER]", filing_type: "10-K", item: "7" } + ``` + For 10-Q, MD&A is item "2": + ``` + action: "get_filing_items" + params: { ticker: "[TICKER]", filing_type: "10-Q", item: "2" } + ``` + +3. **Read Risk Factors**: + ``` + action: "get_filing_items" + params: { ticker: "[TICKER]", filing_type: "10-K", item: "1A" } + ``` + +4. **Extract and analyze**: + - Management's explanation of revenue and margin trends + - Forward-looking statements and guidance + - Key risk factors that could impact financial health + - Any disclosures about accounting policy changes + - Cross-validate: Does management narrative align with the quantitative data from Steps 2-4? + - Flag contradictions between management tone and actual numbers + +5. **Supplement with earnings call transcript** (REQUIRED β€” web search/fetch): + You MUST search for and incorporate the most recent earnings call. This is critical for understanding management's forward-looking view. + - Search for the transcript: + `web_search("[COMPANY] [QUARTER] [YEAR] earnings call transcript")` + - If a transcript URL is found, use `web_fetch` to read key sections (CEO/CFO prepared remarks, Q&A highlights). + - Extract: forward guidance, segment-level commentary, management tone on competitive position, key analyst concerns. + - Cross-reference earnings call statements with MD&A disclosures β€” flag any inconsistencies. + +6. **Summarize key insights**: + - What management says about the business trajectory + - Material risks not visible in the numbers alone + - Any changes in risk factors vs. prior filings (if noticeable) + - Key analyst questions and management responses from earnings call (if available) + +### Step 8: Peer Comparison (Conditional) + +**Execute this step only when the user explicitly requests peer comparison or industry benchmarking.** + +1. **Identify peers**: + - Use the `sector` and `industry` from `get_company_facts` + - Select 2-3 publicly traded competitors in the same industry + - If the user specifies peers, use those instead + +2. **Pull peer data** (for each peer): + ``` + action: "get_financial_metrics_snapshot" + params: { ticker: "[PEER_TICKER]" } + ``` + ``` + action: "get_income_statements" + params: { ticker: "[PEER_TICKER]", period: "annual", limit: 1 } + ``` + ``` + action: "get_balance_sheets" + params: { ticker: "[PEER_TICKER]", period: "annual", limit: 1 } + ``` + +3. **Comparative table**: + + | Metric | [TARGET] | [PEER 1] | [PEER 2] | [PEER 3] | Sector Avg | + |--------|----------|----------|----------|----------|------------| + | Revenue Growth (YoY) | | | | | | + | Gross Margin | | | | | | + | Net Margin | | | | | | + | ROE | | | | | | + | D/E Ratio | | | | | | + | FCF Margin | | | | | | + | P/E Ratio | | | | | | + +4. **Competitive position assessment**: + - Where does the target company rank among peers on each metric? + - Identify clear advantages and disadvantages relative to peers + - Note if the target trades at a premium or discount to peers and whether it's justified + +### Step 9: Present Findings + +Compile the full analysis into a structured report. Follow this exact structure: + +#### 1. Executive Summary +- Company name, ticker, sector, current price +- One-paragraph thesis: Is this a financially healthy company with a durable competitive advantage? +- Financial health rating from Buffett scorecard (Excellent / Good / Average / Weak) +- Earnings quality assessment (High / Moderate / Low) + +#### 2. Financial Health Scorecard +- Full Buffett checklist scorecard table from Step 5 +- Total score and rating + +#### 3. Trend Dashboard +- 5-year key metrics trend table from Steps 2-4: + +| Metric | Y1 | Y2 | Y3 | Y4 | Y5 | Trend | +|--------|----|----|----|----|----|----| +| Revenue | | | | | | arrow | +| Gross Margin | | | | | | arrow | +| Net Margin | | | | | | arrow | +| ROE | | | | | | arrow | +| D/E Ratio | | | | | | arrow | +| FCF | | | | | | arrow | +| OCF/NI | | | | | | arrow | +| CapEx/NI | | | | | | arrow | + +Use directional indicators in the Trend column. + +#### 4. Quality of Earnings +- Summary from Step 6 with key metrics and assessment + +#### 5. Key Strengths & Red Flags +- **Strengths**: List 3-5 financial strengths with supporting data +- **Red Flags**: List any warning signs discovered during analysis. If none, state "No material red flags identified." + +Common red flags to watch for: +- Revenue growth but declining margins +- Net income growing but OCF declining +- AR growing faster than revenue +- Inventory building up vs. flat COGS +- Rising debt with declining interest coverage +- Retained earnings declining +- Large goodwill relative to total assets +- CapEx consistently > 50% of net income +- Management tone in MD&A contradicts financial data + +#### 6. SEC Filing Insights +- Key findings from Step 7 +- Management's outlook and material risks + +#### 7. Peer Comparison (if Step 8 was executed) +- Comparative table and competitive position assessment + +### Guardrails + +- Always state the date range of financial data used. +- If any data is missing or unavailable, explicitly note it and adjust the analysis scope. +- Do not present calculated ratios as precise β€” round to one decimal place. +- Clearly distinguish between facts (from data) and interpretive conclusions. +- The Buffett scorecard is a screening framework, not a buy/sell recommendation. State this in the output. +- For non-US companies or companies not filing with the SEC, skip Step 7 and note the limitation. +- Output language must match the user's input language (Chinese input β†’ Chinese output, English input β†’ English output). + +### Web Search Requirements + +**Minimum mandatory searches (you MUST perform these):** +1. Earnings call highlights (Step 1b) β€” for management's own explanation of results +2. Industry outlook (Step 1b) β€” for macro/sector context +3. Earnings call transcript (Step 7) β€” for forward guidance and analyst Q&A + +**Additional searches (trigger when data shows anomalies):** +- Revenue or margin inflection points (Steps 2-4) +- Major debt changes or acquisitions (Step 3) +- CapEx spikes (Step 4) +- Quality-of-earnings red flags (Step 6) + +**Search principles:** +- **Source quality**: Prefer primary sources (SEC filings, company press releases, earnings call transcripts) over secondary sources (analyst blogs, news aggregators). +- **Cite with dates**: Always include source name and date when referencing external information. +- **Separate fact from opinion**: Label analyst or media commentary as external opinion, not fact. +- **Total budget**: Expect 3-8 web searches per analysis. Fewer than 3 means you are likely missing critical context. diff --git a/skills/earnings-analysis/references/buffett-checklist.md b/skills/earnings-analysis/references/buffett-checklist.md new file mode 100644 index 00000000..ebab5bae --- /dev/null +++ b/skills/earnings-analysis/references/buffett-checklist.md @@ -0,0 +1,99 @@ +# Buffett Competitive Advantage Checklist + +Score each criterion and calculate a total. Use this to assess whether a company has a durable competitive advantage (economic moat). + +## Scoring System + +Total: 100 points across 4 categories (25 points each). + +### Category 1: Profitability (25 points) + +| # | Criterion | Excellent | Good | Weak | +|---|-----------|-----------|------|------| +| 1 | **Gross Margin** | > 40% β†’ **10 pts** | 30-40% β†’ **6 pts** | < 30% β†’ **2 pts** | +| 2 | **Net Margin** | > 20% β†’ **10 pts** | 10-20% β†’ **6 pts** | < 10% β†’ **2 pts** | +| 3 | **Return on Equity (ROE)** | > 15% β†’ **5 pts** | 10-15% β†’ **3 pts** | < 10% β†’ **1 pt** | + +How to calculate: +- Gross Margin = Gross Profit / Revenue +- Net Margin = Net Income / Revenue +- ROE = Net Income / Total Shareholders' Equity +- Use the most recent annual figures; cross-check with 5-year average + +### Category 2: Balance Sheet Health (25 points) + +| # | Criterion | Pass | Partial | Fail | +|---|-----------|------|---------|------| +| 4 | **Cash > Total Debt** | Yes β†’ **8 pts** | Cash > 50% of Debt β†’ **4 pts** | Cash < 50% of Debt β†’ **1 pt** | +| 5 | **Debt-to-Equity Ratio** | < 0.8 β†’ **7 pts** | 0.8-1.5 β†’ **4 pts** | > 1.5 β†’ **1 pt** | +| 6 | **No Preferred Stock** | None β†’ **5 pts** | β€” | Has Preferred β†’ **0 pts** | +| 7 | **Retained Earnings Growth** | Growing 5 consecutive years β†’ **5 pts** | Growing 3-4 years β†’ **3 pts** | Declining or flat β†’ **1 pt** | + +How to calculate: +- Cash = Cash and Cash Equivalents + Short-term Investments +- Total Debt = Short-term Debt + Long-term Debt +- D/E = Total Liabilities / Total Shareholders' Equity +- Retained Earnings: Compare year-over-year from balance sheets + +Special note on D/E: +- Exclude operating lease liabilities from "debt" for this assessment (they are contractual obligations, not financial debt) +- If treasury stock is large, it reduces equity and inflates D/E β€” note this in analysis + +### Category 3: Cash Flow Quality (25 points) + +| # | Criterion | Excellent | Good | Weak | +|---|-----------|-----------|------|------| +| 8 | **CapEx / Net Income** | < 25% β†’ **10 pts** | 25-50% β†’ **6 pts** | > 50% β†’ **2 pts** | +| 9 | **Operating CF > Net Income** | OCF/NI > 1.0 β†’ **8 pts** | OCF/NI = 0.8-1.0 β†’ **4 pts** | OCF/NI < 0.8 β†’ **1 pt** | +| 10 | **Shareholder Returns** | Buybacks + Dividends β†’ **7 pts** | Dividends only β†’ **4 pts** | Neither β†’ **1 pt** | + +How to calculate: +- CapEx: Capital Expenditure from cash flow statement (use absolute value) +- Operating CF: Net Cash from Operating Activities +- Buybacks: Check if Treasury Stock increased year-over-year, or look at "repurchase of common stock" in financing activities +- Dividends: Look at "dividends paid" in financing activities + +Note on CapEx: +- One-time large CapEx (e.g., new factory, data center buildout) should be noted but not penalized if the 5-year average CapEx/NI is still within range +- Asset-light businesses (software, services) naturally score well here + +### Category 4: Consistency (25 points) + +| # | Criterion | Excellent | Good | Weak | +|---|-----------|-----------|------|------| +| 11 | **Revenue Growth Streak** | 5+ consecutive years growing β†’ **10 pts** | 3-4 years β†’ **6 pts** | < 3 years β†’ **2 pts** | +| 12 | **Net Income Growth Streak** | 5+ consecutive years growing β†’ **10 pts** | 3-4 years β†’ **6 pts** | < 3 years β†’ **2 pts** | +| 13 | **Recession Resilience** | Profitable through last recession β†’ **5 pts** | Revenue dip < 10% β†’ **3 pts** | Significant losses β†’ **1 pt** | + +How to assess: +- Revenue/NI growth: Check year-over-year changes for the last 5 years +- Recession resilience: Check 2020 (COVID) and 2022 (rate hikes) performance. For older data, check 2008-2009 if available. +- A single flat year in an otherwise consistent growth streak can be scored as "Good" + +## Score Interpretation + +| Total Score | Rating | Interpretation | +|-------------|--------|----------------| +| 80-100 | **Excellent** | Strong durable competitive advantage. Consistent profitability, fortress balance sheet, capital-light operations. Classic Buffett-style investment candidate. | +| 60-79 | **Good** | Solid business with some competitive advantages. May have minor weaknesses in one category. Worth deeper investigation. | +| 40-59 | **Average** | Mediocre competitive position. Multiple areas of concern. Higher risk of margin erosion or competitive disruption. | +| < 40 | **Weak** | No clear competitive advantage. High debt, inconsistent earnings, or capital-intensive operations. Not a typical Buffett investment. | + +## Sector-Specific Caveats + +- **Financials**: Skip gross margin (criterion 1). Use net interest margin > 3% as substitute for 10 pts. D/E ratio thresholds don't apply β€” use Tier 1 Capital Ratio > 10% for 7 pts instead. +- **Utilities**: Naturally capital-intensive (CapEx criterion will score low). Offset by checking regulated return stability. If regulated ROE is consistently 9-11%, award 6 pts for criterion 8. +- **REITs**: Required to pay out 90%+ as dividends, so retained earnings won't grow. Skip criterion 7; award 5 pts if FFO per share grows consistently instead. +- **Growth-stage Tech**: May not yet have 5 years of profitability. Score consistency based on revenue growth and gross margin expansion trajectory. Note that the overall score may be artificially low. + +## Output Format + +Present the scorecard as a table: + +| # | Criterion | Value | Score | Max | +|---|-----------|-------|-------|-----| +| 1 | Gross Margin | 43.2% | 10 | 10 | +| 2 | Net Margin | 25.1% | 10 | 10 | +| ... | ... | ... | ... | ... | +| | **Total** | | **XX** | **100** | +| | **Rating** | | **Excellent/Good/Average/Weak** | | diff --git a/skills/earnings-analysis/references/financial-ratios-benchmarks.md b/skills/earnings-analysis/references/financial-ratios-benchmarks.md new file mode 100644 index 00000000..5e0b537e --- /dev/null +++ b/skills/earnings-analysis/references/financial-ratios-benchmarks.md @@ -0,0 +1,70 @@ +# Financial Ratios Benchmarks by Sector + +Use the company's `sector` from `get_company_facts` to look up benchmark ranges below. Compare the company's ratios against these benchmarks and note deviations. + +## Profitability Benchmarks + +| Sector | Gross Margin | Operating Margin | Net Margin | ROE | ROA | +|--------|-------------|-----------------|------------|-----|-----| +| Communication Services | 50-60% | 15-25% | 10-18% | 12-20% | 5-10% | +| Consumer Discretionary | 35-50% | 8-15% | 5-10% | 15-25% | 5-10% | +| Consumer Staples | 35-45% | 12-18% | 8-12% | 20-30% | 8-12% | +| Energy | 30-50% | 10-20% | 5-15% | 10-20% | 5-10% | +| Financials | N/A | 25-35% | 15-25% | 10-15% | 1-2% | +| Health Care | 55-70% | 15-25% | 10-20% | 15-25% | 8-12% | +| Industrials | 25-35% | 10-15% | 6-10% | 15-20% | 5-8% | +| Information Technology | 55-70% | 20-30% | 15-25% | 20-35% | 10-15% | +| Materials | 25-35% | 10-18% | 5-12% | 10-18% | 5-8% | +| Real Estate | 55-70% | 25-40% | 15-30% | 5-10% | 2-5% | +| Utilities | 35-50% | 15-25% | 8-15% | 8-12% | 3-5% | + +## Balance Sheet Benchmarks + +| Sector | Current Ratio | Quick Ratio | D/E Ratio | Interest Coverage | +|--------|--------------|-------------|-----------|-------------------| +| Communication Services | 1.0-1.5 | 0.8-1.2 | 0.8-1.5 | 4-8x | +| Consumer Discretionary | 1.2-2.0 | 0.8-1.5 | 0.5-1.2 | 5-10x | +| Consumer Staples | 1.0-1.5 | 0.6-1.0 | 0.5-1.0 | 8-15x | +| Energy | 1.0-1.5 | 0.8-1.2 | 0.3-0.8 | 5-10x | +| Financials | N/A | N/A | 2.0-8.0 | N/A | +| Health Care | 1.5-2.5 | 1.2-2.0 | 0.3-0.8 | 8-15x | +| Industrials | 1.2-2.0 | 0.8-1.5 | 0.5-1.0 | 6-12x | +| Information Technology | 2.0-3.5 | 1.5-3.0 | 0.2-0.6 | 15-30x | +| Materials | 1.5-2.5 | 1.0-1.5 | 0.4-0.8 | 6-12x | +| Real Estate | 1.0-1.5 | 0.5-1.0 | 0.8-1.5 | 3-5x | +| Utilities | 0.8-1.2 | 0.5-0.8 | 1.0-2.0 | 3-5x | + +## Cash Flow Benchmarks + +| Sector | FCF Margin | CapEx/Revenue | Op. CF / Net Income | +|--------|-----------|---------------|---------------------| +| Communication Services | 10-20% | 10-20% | 1.2-1.8x | +| Consumer Discretionary | 5-12% | 3-8% | 1.1-1.5x | +| Consumer Staples | 8-15% | 3-6% | 1.2-1.5x | +| Energy | 5-15% | 15-30% | 1.5-2.5x | +| Financials | N/A | 1-3% | N/A | +| Health Care | 15-25% | 3-8% | 1.2-1.8x | +| Industrials | 5-12% | 3-8% | 1.2-1.6x | +| Information Technology | 20-35% | 3-10% | 1.2-1.8x | +| Materials | 5-12% | 5-12% | 1.3-2.0x | +| Real Estate | 15-30% | 5-15% | 1.5-3.0x | +| Utilities | 5-10% | 15-25% | 2.0-3.5x | + +## Usage Notes + +- **Financials sector**: Gross margin and current/quick ratios are not meaningful for banks and insurers. Use net interest margin and capital adequacy ratios instead. +- **Real Estate**: High depreciation makes net margin less useful. Focus on Funds From Operations (FFO). +- **Growth-stage companies**: May have negative margins. Compare against growth-stage peers rather than mature sector benchmarks. +- **Cyclical sectors** (Energy, Materials, Industrials): Use cycle-average margins (5-7 years) rather than single-year comparisons. +- **Post-M&A**: Goodwill and amortization may distort margins for 1-2 years after acquisitions. Note any large acquisitions. + +## Buffett's Rules of Thumb (Quick Reference) + +| Metric | Excellent | Good | Weak | +|--------|-----------|------|------| +| Gross Margin | > 40% | 30-40% | < 30% | +| Net Margin | > 20% | 10-20% | < 10% | +| ROE | > 15% | 10-15% | < 10% | +| D/E Ratio | < 0.5 | 0.5-0.8 | > 0.8 | +| CapEx / Net Income | < 25% | 25-50% | > 50% | +| Debt Payoff (years) | < 2 | 2-4 | > 4 | diff --git a/skills/finance-research/SKILL.md b/skills/finance-research/SKILL.md new file mode 100644 index 00000000..1aae38dc --- /dev/null +++ b/skills/finance-research/SKILL.md @@ -0,0 +1,171 @@ +--- +name: Finance Research +description: Conduct analyst-grade financial research across primary and secondary markets using structured financial data plus macro and public-information cross-checks. +version: 1.1.1 +metadata: + emoji: "\U0001F4CA" + tags: + - finance + - research + - stocks + - data + - macro + - sentiment +userInvocable: true +disableModelInvocation: false +--- + +## Instructions + +You are conducting financial research with an analyst-grade standard. Tool usage is a dynamic decision. Do not force tool combinations. Choose tools based on evidence sufficiency for the specific question. + +### Available Data Actions + +#### Price Data +- `get_price_snapshot` β€” Current stock price. Params: `{ ticker }` +- `get_prices` β€” Historical OHLCV prices. Params: `{ ticker, start_date, end_date, interval?, interval_multiplier? }` + - interval: "day" (default), "week", "month", "year" +- `get_crypto_price_snapshot` β€” Current crypto price. Params: `{ ticker }` (e.g. "BTC-USD") +- `get_crypto_prices` β€” Historical crypto prices. Same params as get_prices. +- `get_available_crypto_tickers` β€” List available crypto tickers. Params: `{}` + +#### Financial Statements +All share params: `{ ticker, period, limit?, report_period_gt?, report_period_gte?, report_period_lt?, report_period_lte? }` +- period: "annual", "quarterly", or "ttm" +- Dates in YYYY-MM-DD format + +Actions: +- `get_income_statements` β€” Revenue, expenses, net income, EPS +- `get_balance_sheets` β€” Assets, liabilities, equity, debt, cash +- `get_cash_flow_statements` β€” Operating, investing, financing cash flows, FCF +- `get_all_financial_statements` β€” All three at once (more efficient when you need multiple) + +#### Metrics & Estimates +- `get_financial_metrics_snapshot` β€” Current key ratios (P/E, market cap, margins, etc.). Params: `{ ticker }` +- `get_financial_metrics` β€” Historical metrics. Params: `{ ticker, period?, limit?, report_period*? }` +- `get_analyst_estimates` β€” EPS and revenue estimates. Params: `{ ticker, period? }` + +#### Company Info +- `get_company_facts` β€” Sector, industry, employees, exchange, website. Params: `{ ticker }` +- `get_news` β€” Recent company news articles. Params: `{ ticker, start_date?, end_date?, limit? }` +- `get_insider_trades` β€” Insider buying/selling (SEC Form 4). Params: `{ ticker, limit?, filing_date*? }` +- `get_segmented_revenues` β€” Revenue by segment/geography. Params: `{ ticker, period, limit? }` + +#### SEC Filings +- `get_filings` β€” List filings metadata. Params: `{ ticker, filing_type?, limit? }` +- `get_filing_items` β€” Read filing sections. Params: `{ ticker, filing_type, accession_number?, item? }` + +### Evidence Sufficiency Gate (Internal Decision) + +Before deep analysis, make an internal evidence decision. Do not output a technical decision block by default. + +If the user explicitly asks for methodology or reasoning transparency, provide a concise plain-language explanation of your research approach. + +Decision policy: + +- Start with `data_only` when structured data can support the requested conclusion. +- Escalate to `hybrid` when the task is event-driven, time-sensitive, or requires causal explanation not visible in structured data alone. +- Use `web_first` only when the task is mainly document/news/policy driven (common in pre-IPO without stable ticker coverage). +- If a tool is unavailable, continue with available tools and explicitly downgrade confidence. + +### Core Analysis Framework + +1. **Scope & Market Type** +- Identify if this is primary market (IPO, pre-IPO, follow-on, placement) or secondary market (listed stock/sector/index). +- State region and analysis horizon (event-driven, 3-6 months, 1-3 years). + +2. **Core Company Data (Structured)** +- Start with: `get_price_snapshot`, `get_company_facts`, `get_financial_metrics_snapshot`. +- Pull statements (`get_all_financial_statements`) and estimates as needed. + +3. **Macro & Policy Context (Conditional)** +- Use `web_search` / `web_fetch` only if required by your internal evidence decision. +- If used, prefer high-signal primary sources (central bank, regulator, official releases). +- For time-sensitive conclusions, include source dates explicitly. + +4. **News & Sentiment Context (Conditional)** +- Use `get_news` for company-linked coverage when available. +- Add web cross-checks only when event validation materially affects the conclusion. + +5. **Synthesis & Decision** +- Separate **facts**, **inference**, and **assumptions**. +- Build bull/base/bear scenarios with explicit trigger conditions. +- Provide confidence level and explain the main uncertainty drivers. + +### Primary Market (δΈ€ηΊ§εΈ‚εœΊ) Workflow + +When asked about IPOs, pre-IPO, or new issuance: + +1. **Deal Basics** +- Identify issuer, listing venue, offering structure (primary/secondary shares), expected timeline. +- Determine whether a reliable ticker exists in current data coverage. + +2. **Filing/Prospectus Review** +- Prefer official documents (e.g., S-1/F-1/prospectus) via `web_search` + `web_fetch`. +- Extract: use of proceeds, customer concentration, related-party transactions, share classes, lock-up, dilution risks. + +Primary-market capability boundary: +- If `ticker` is available and filings are retrievable, run hybrid analysis (structured + document evidence). +- If `ticker` is unavailable or structured filing fields are limited, run web-led analysis and clearly label it as partial-coverage with reduced confidence. + +3. **Valuation & Comparable Set** +- Build peer set from listed comps (secondary market tickers) and compare growth, margin, and valuation multiples. +- Flag gaps between issuer narrative and peer reality. + +4. **Deal Risk Map** +- Highlight red flags: weak FCF quality, aggressive non-GAAP adjustments, concentrated revenue, regulatory overhang. +- Provide post-listing watch items: lock-up expiry, first earnings, guidance revisions. + +### Secondary Market (δΊŒηΊ§εΈ‚εœΊ) Workflow + +When asked about listed equities: + +1. **Trend & Positioning** +- Pull 1y price history (`get_prices`) and identify regime (uptrend/range/downtrend) with volatility context. + +2. **Fundamentals** +- Analyze growth quality (revenue vs FCF), margin durability, leverage, and capital allocation. + +3. **Valuation** +- Compare current multiples to historical bands and peers (when peer data is available). +- Connect valuation premium/discount to expected growth and risk profile. + +4. **Catalysts & Risks** +- Earnings, guidance, product cycle, policy changes, rates/FX/commodity sensitivity, insider activity. + +### Output Standard + +Always include: + +1. **Executive Summary** (thesis + stance + confidence) +2. **Evidence Table** with columns: +- Signal +- Direction (Bull/Bear/Neutral) +- Why it matters +- Source +- Date +3. **Scenario Table** (bull/base/bear with probabilities or relative weights) +4. **Key Monitoring Triggers** (what would invalidate current thesis) + +### Guardrails + +- Always state data cutoff dates. +- If data is missing, explicitly mark it and show the impact on confidence. +- Do not present assumptions as facts. +- For event-driven conclusions, if you skip web validation, explicitly explain why structured evidence is still sufficient. + + +### Example: Secondary Market Analysis + +For "Analyze Apple's investment outlook": + +1. `data(domain="finance", action="get_price_snapshot", params={ticker: "AAPL"})` +2. `data(domain="finance", action="get_company_facts", params={ticker: "AAPL"})` +3. `data(domain="finance", action="get_all_financial_statements", params={ticker: "AAPL", period: "annual", limit: 3})` +4. `data(domain="finance", action="get_financial_metrics", params={ticker: "AAPL", period: "quarterly", limit: 8})` +5. `data(domain="finance", action="get_analyst_estimates", params={ticker: "AAPL", period: "annual"})` +6. `data(domain="finance", action="get_news", params={ticker: "AAPL", limit: 10})` +7. `web_search(query="latest Fed policy decision impact on US mega-cap tech valuations")` +8. `web_search(query="Apple supply chain or regulatory news latest quarter")` + +Then synthesize fundamental trend, macro regime, and event sentiment into a scenario-based conclusion. diff --git a/skills/pdf/SKILL.md b/skills/pdf/SKILL.md new file mode 100644 index 00000000..c35eb4ee --- /dev/null +++ b/skills/pdf/SKILL.md @@ -0,0 +1,335 @@ +--- +name: PDF Processing +description: Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill. +version: 1.0.0 +metadata: + emoji: "πŸ“•" + tags: + - office + - document + - pdf + install: + - id: brew-poppler + kind: brew + formula: poppler + bins: [pdftoppm, pdftotext, pdfimages] + label: "Install poppler for PDF text/image extraction" + os: [darwin, linux] + - id: brew-qpdf + kind: brew + formula: qpdf + bins: [qpdf] + label: "Install qpdf for advanced PDF manipulation" + os: [darwin, linux] +userInvocable: true +disableModelInvocation: false +--- + +# PDF Processing Guide + +## Overview + +This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see reference.md. If you need to fill out a PDF form, read forms.md and follow its instructions. + +## Quick Start + +```python +from pypdf import PdfReader, PdfWriter + +# Read a PDF +reader = PdfReader("document.pdf") +print(f"Pages: {len(reader.pages)}") + +# Extract text +text = "" +for page in reader.pages: + text += page.extract_text() +``` + +## Python Libraries + +### pypdf - Basic Operations + +#### Merge PDFs +```python +from pypdf import PdfWriter, PdfReader + +writer = PdfWriter() +for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]: + reader = PdfReader(pdf_file) + for page in reader.pages: + writer.add_page(page) + +with open("merged.pdf", "wb") as output: + writer.write(output) +``` + +#### Split PDF +```python +reader = PdfReader("input.pdf") +for i, page in enumerate(reader.pages): + writer = PdfWriter() + writer.add_page(page) + with open(f"page_{i+1}.pdf", "wb") as output: + writer.write(output) +``` + +#### Extract Metadata +```python +reader = PdfReader("document.pdf") +meta = reader.metadata +print(f"Title: {meta.title}") +print(f"Author: {meta.author}") +print(f"Subject: {meta.subject}") +print(f"Creator: {meta.creator}") +``` + +#### Rotate Pages +```python +reader = PdfReader("input.pdf") +writer = PdfWriter() + +page = reader.pages[0] +page.rotate(90) # Rotate 90 degrees clockwise +writer.add_page(page) + +with open("rotated.pdf", "wb") as output: + writer.write(output) +``` + +### pdfplumber - Text and Table Extraction + +#### Extract Text with Layout +```python +import pdfplumber + +with pdfplumber.open("document.pdf") as pdf: + for page in pdf.pages: + text = page.extract_text() + print(text) +``` + +#### Extract Tables +```python +with pdfplumber.open("document.pdf") as pdf: + for i, page in enumerate(pdf.pages): + tables = page.extract_tables() + for j, table in enumerate(tables): + print(f"Table {j+1} on page {i+1}:") + for row in table: + print(row) +``` + +#### Advanced Table Extraction +```python +import pandas as pd + +with pdfplumber.open("document.pdf") as pdf: + all_tables = [] + for page in pdf.pages: + tables = page.extract_tables() + for table in tables: + if table: # Check if table is not empty + df = pd.DataFrame(table[1:], columns=table[0]) + all_tables.append(df) + +# Combine all tables +if all_tables: + combined_df = pd.concat(all_tables, ignore_index=True) + combined_df.to_excel("extracted_tables.xlsx", index=False) +``` + +### reportlab - Create PDFs + +#### Basic PDF Creation +```python +from reportlab.lib.pagesizes import letter +from reportlab.pdfgen import canvas + +c = canvas.Canvas("hello.pdf", pagesize=letter) +width, height = letter + +# Add text +c.drawString(100, height - 100, "Hello World!") +c.drawString(100, height - 120, "This is a PDF created with reportlab") + +# Add a line +c.line(100, height - 140, 400, height - 140) + +# Save +c.save() +``` + +#### Create PDF with Multiple Pages +```python +from reportlab.lib.pagesizes import letter +from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak +from reportlab.lib.styles import getSampleStyleSheet + +doc = SimpleDocTemplate("report.pdf", pagesize=letter) +styles = getSampleStyleSheet() +story = [] + +# Add content +title = Paragraph("Report Title", styles['Title']) +story.append(title) +story.append(Spacer(1, 12)) + +body = Paragraph("This is the body of the report. " * 20, styles['Normal']) +story.append(body) +story.append(PageBreak()) + +# Page 2 +story.append(Paragraph("Page 2", styles['Heading1'])) +story.append(Paragraph("Content for page 2", styles['Normal'])) + +# Build PDF +doc.build(story) +``` + +#### Subscripts and Superscripts + +**IMPORTANT**: Never use Unicode subscript/superscript characters in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes. + +Instead, use ReportLab's XML markup tags in Paragraph objects: +```python +from reportlab.platypus import Paragraph +from reportlab.lib.styles import getSampleStyleSheet + +styles = getSampleStyleSheet() + +# Subscripts: use tag +chemical = Paragraph("H2O", styles['Normal']) + +# Superscripts: use tag +squared = Paragraph("x2 + y2", styles['Normal']) +``` + +For canvas-drawn text (not Paragraph objects), manually adjust font the size and position rather than using Unicode subscripts/superscripts. + +## Command-Line Tools + +### pdftotext (poppler-utils) +```bash +# Extract text +pdftotext input.pdf output.txt + +# Extract text preserving layout +pdftotext -layout input.pdf output.txt + +# Extract specific pages +pdftotext -f 1 -l 5 input.pdf output.txt # Pages 1-5 +``` + +### qpdf +```bash +# Merge PDFs +qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf + +# Split pages +qpdf input.pdf --pages . 1-5 -- pages1-5.pdf +qpdf input.pdf --pages . 6-10 -- pages6-10.pdf + +# Rotate pages +qpdf input.pdf output.pdf --rotate=+90:1 # Rotate page 1 by 90 degrees + +# Remove password +qpdf --password=mypassword --decrypt encrypted.pdf decrypted.pdf +``` + +### pdftk (if available) +```bash +# Merge +pdftk file1.pdf file2.pdf cat output merged.pdf + +# Split +pdftk input.pdf burst + +# Rotate +pdftk input.pdf rotate 1east output rotated.pdf +``` + +## Common Tasks + +### Extract Text from Scanned PDFs +```python +# Requires: pip install pytesseract pdf2image +import pytesseract +from pdf2image import convert_from_path + +# Convert PDF to images +images = convert_from_path('scanned.pdf') + +# OCR each page +text = "" +for i, image in enumerate(images): + text += f"Page {i+1}:\n" + text += pytesseract.image_to_string(image) + text += "\n\n" + +print(text) +``` + +### Add Watermark +```python +from pypdf import PdfReader, PdfWriter + +# Create watermark (or load existing) +watermark = PdfReader("watermark.pdf").pages[0] + +# Apply to all pages +reader = PdfReader("document.pdf") +writer = PdfWriter() + +for page in reader.pages: + page.merge_page(watermark) + writer.add_page(page) + +with open("watermarked.pdf", "wb") as output: + writer.write(output) +``` + +### Extract Images +```bash +# Using pdfimages (poppler-utils) +pdfimages -j input.pdf output_prefix + +# This extracts all images as output_prefix-000.jpg, output_prefix-001.jpg, etc. +``` + +### Password Protection +```python +from pypdf import PdfReader, PdfWriter + +reader = PdfReader("input.pdf") +writer = PdfWriter() + +for page in reader.pages: + writer.add_page(page) + +# Add password +writer.encrypt("userpassword", "ownerpassword") + +with open("encrypted.pdf", "wb") as output: + writer.write(output) +``` + +## Quick Reference + +| Task | Best Tool | Command/Code | +|------|-----------|--------------| +| Merge PDFs | pypdf | `writer.add_page(page)` | +| Split PDFs | pypdf | One page per file | +| Extract text | pdfplumber | `page.extract_text()` | +| Extract tables | pdfplumber | `page.extract_tables()` | +| Create PDFs | reportlab | Canvas or Platypus | +| Command line merge | qpdf | `qpdf --empty --pages ...` | +| OCR scanned PDFs | pytesseract | Convert to image first | +| Fill PDF forms | pdf-lib or pypdf (see forms.md) | See forms.md | + +## Next Steps + +- For advanced pypdfium2 usage, see reference.md +- For JavaScript libraries (pdf-lib), see reference.md +- If you need to fill out a PDF form, follow the instructions in forms.md +- For troubleshooting guides, see reference.md diff --git a/skills/pdf/forms.md b/skills/pdf/forms.md new file mode 100644 index 00000000..2dece27c --- /dev/null +++ b/skills/pdf/forms.md @@ -0,0 +1,294 @@ +**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.** + +If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory: + `python scripts/extract_form_field_info.py --check `, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions. + +# Fillable fields +If the PDF has fillable form fields: +- Run this script from this file's directory: `python scripts/extract_form_field_info.py `. It will create a JSON file with a list of fields in this format: +``` +[ + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page), + "type": ("text", "checkbox", "radio_group", or "choice"), + }, + // Checkboxes have "checked_value" and "unchecked_value" properties: + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "checkbox", + "checked_value": (Set the field to this value to check the checkbox), + "unchecked_value": (Set the field to this value to uncheck the checkbox), + }, + // Radio groups have a "radio_options" list with the possible choices. + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "radio_group", + "radio_options": [ + { + "value": (set the field to this value to select this radio option), + "rect": (bounding box for the radio button for this option) + }, + // Other radio options + ] + }, + // Multiple choice fields have a "choice_options" list with the possible choices: + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "choice", + "choice_options": [ + { + "value": (set the field to this value to select this option), + "text": (display text of the option) + }, + // Other choice options + ], + } +] +``` +- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory): +`python scripts/convert_pdf_to_images.py ` +Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates). +- Create a `field_values.json` file in this format with the values to be entered for each field: +``` +[ + { + "field_id": "last_name", // Must match the field_id from `extract_form_field_info.py` + "description": "The user's last name", + "page": 1, // Must match the "page" value in field_info.json + "value": "Simpson" + }, + { + "field_id": "Checkbox12", + "description": "Checkbox to be checked if the user is 18 or over", + "page": 1, + "value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options". + }, + // more fields +] +``` +- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF: +`python scripts/fill_fillable_fields.py ` +This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again. + +# Non-fillable fields +If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed. + +## Step 1: Try Structure Extraction First + +Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates: +`python scripts/extract_form_structure.py form_structure.json` + +This creates a JSON file containing: +- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points) +- **lines**: Horizontal lines that define row boundaries +- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates) +- **row_boundaries**: Row top/bottom positions calculated from horizontal lines + +**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**. + +--- + +## Approach A: Structure-Based Coordinates (Preferred) + +Use this when `extract_form_structure.py` found text labels in the PDF. + +### A.1: Analyze the Structure + +Read form_structure.json and identify: + +1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name") +2. **Row structure**: Labels with similar `top` values are in the same row +3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap) +4. **Checkboxes**: Use the checkbox coordinates directly from the structure + +**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward. + +### A.2: Check for Missing Elements + +The structure extraction may not detect all form elements. Common cases: +- **Circular checkboxes**: Only square rectangles are detected as checkboxes +- **Complex graphics**: Decorative elements or non-standard form controls +- **Faded or light-colored elements**: May not be extracted + +If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below). + +### A.3: Create fields.json with PDF Coordinates + +For each field, calculate entry coordinates from the extracted structure: + +**Text fields:** +- entry x0 = label x1 + 5 (small gap after label) +- entry x1 = next label's x0, or row boundary +- entry top = same as label top +- entry bottom = row boundary line below, or label bottom + row_height + +**Checkboxes:** +- Use the checkbox rectangle coordinates directly from form_structure.json +- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom] + +Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates): +```json +{ + "pages": [ + {"page_number": 1, "pdf_width": 612, "pdf_height": 792} + ], + "form_fields": [ + { + "page_number": 1, + "description": "Last name entry field", + "field_label": "Last Name", + "label_bounding_box": [43, 63, 87, 73], + "entry_bounding_box": [92, 63, 260, 79], + "entry_text": {"text": "Smith", "font_size": 10} + }, + { + "page_number": 1, + "description": "US Citizen Yes checkbox", + "field_label": "Yes", + "label_bounding_box": [260, 200, 280, 210], + "entry_bounding_box": [285, 197, 292, 205], + "entry_text": {"text": "X"} + } + ] +} +``` + +**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json. + +### A.4: Validate Bounding Boxes + +Before filling, check your bounding boxes for errors: +`python scripts/check_bounding_boxes.py fields.json` + +This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling. + +--- + +## Approach B: Visual Estimation (Fallback) + +Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns). + +### B.1: Convert PDF to Images + +`python scripts/convert_pdf_to_images.py ` + +### B.2: Initial Field Identification + +Examine each page image to identify form sections and get **rough estimates** of field locations: +- Form field labels and their approximate positions +- Entry areas (lines, boxes, or blank spaces for text input) +- Checkboxes and their approximate locations + +For each field, note approximate pixel coordinates (they don't need to be precise yet). + +### B.3: Zoom Refinement (CRITICAL for accuracy) + +For each field, crop a region around the estimated position to refine coordinates precisely. + +**Create a zoomed crop using ImageMagick:** +```bash +magick -crop x++ +repage +``` + +Where: +- `, ` = top-left corner of crop region (use your rough estimate minus padding) +- `, ` = size of crop region (field area plus ~50px padding on each side) + +**Example:** To refine a "Name" field estimated around (100, 150): +```bash +magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png +``` + +(Note: if the `magick` command isn't available, try `convert` with the same arguments). + +**Examine the cropped image** to determine precise coordinates: +1. Identify the exact pixel where the entry area begins (after the label) +2. Identify where the entry area ends (before next field or edge) +3. Identify the top and bottom of the entry line/box + +**Convert crop coordinates back to full image coordinates:** +- full_x = crop_x + crop_offset_x +- full_y = crop_y + crop_offset_y + +Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop: +- entry_x0 = 52 + 50 = 102 +- entry_top = 18 + 120 = 138 + +**Repeat for each field**, grouping nearby fields into single crops when possible. + +### B.4: Create fields.json with Refined Coordinates + +Create fields.json using `image_width` and `image_height` (signals image coordinates): +```json +{ + "pages": [ + {"page_number": 1, "image_width": 1700, "image_height": 2200} + ], + "form_fields": [ + { + "page_number": 1, + "description": "Last name entry field", + "field_label": "Last Name", + "label_bounding_box": [120, 175, 242, 198], + "entry_bounding_box": [255, 175, 720, 218], + "entry_text": {"text": "Smith", "font_size": 10} + } + ] +} +``` + +**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis. + +### B.5: Validate Bounding Boxes + +Before filling, check your bounding boxes for errors: +`python scripts/check_bounding_boxes.py fields.json` + +This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling. + +--- + +## Hybrid Approach: Structure + Visual + +Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls). + +1. **Use Approach A** for fields that were detected in form_structure.json +2. **Convert PDF to images** for visual analysis of missing fields +3. **Use zoom refinement** (from Approach B) for the missing fields +4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates: + - pdf_x = image_x * (pdf_width / image_width) + - pdf_y = image_y * (pdf_height / image_height) +5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height` + +--- + +## Step 2: Validate Before Filling + +**Always validate bounding boxes before filling:** +`python scripts/check_bounding_boxes.py fields.json` + +This checks for: +- Intersecting bounding boxes (which would cause overlapping text) +- Entry boxes that are too small for the specified font size + +Fix any reported errors in fields.json before proceeding. + +## Step 3: Fill the Form + +The fill script auto-detects the coordinate system and handles conversion: +`python scripts/fill_pdf_form_with_annotations.py fields.json ` + +## Step 4: Verify Output + +Convert the filled PDF to images and verify text placement: +`python scripts/convert_pdf_to_images.py ` + +If text is mispositioned: +- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height` +- **Approach B**: Check that image dimensions match and coordinates are accurate pixels +- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields diff --git a/skills/pdf/reference.md b/skills/pdf/reference.md new file mode 100644 index 00000000..41400bf4 --- /dev/null +++ b/skills/pdf/reference.md @@ -0,0 +1,612 @@ +# PDF Processing Advanced Reference + +This document contains advanced PDF processing features, detailed examples, and additional libraries not covered in the main skill instructions. + +## pypdfium2 Library (Apache/BSD License) + +### Overview +pypdfium2 is a Python binding for PDFium (Chromium's PDF library). It's excellent for fast PDF rendering, image generation, and serves as a PyMuPDF replacement. + +### Render PDF to Images +```python +import pypdfium2 as pdfium +from PIL import Image + +# Load PDF +pdf = pdfium.PdfDocument("document.pdf") + +# Render page to image +page = pdf[0] # First page +bitmap = page.render( + scale=2.0, # Higher resolution + rotation=0 # No rotation +) + +# Convert to PIL Image +img = bitmap.to_pil() +img.save("page_1.png", "PNG") + +# Process multiple pages +for i, page in enumerate(pdf): + bitmap = page.render(scale=1.5) + img = bitmap.to_pil() + img.save(f"page_{i+1}.jpg", "JPEG", quality=90) +``` + +### Extract Text with pypdfium2 +```python +import pypdfium2 as pdfium + +pdf = pdfium.PdfDocument("document.pdf") +for i, page in enumerate(pdf): + text = page.get_text() + print(f"Page {i+1} text length: {len(text)} chars") +``` + +## JavaScript Libraries + +### pdf-lib (MIT License) + +pdf-lib is a powerful JavaScript library for creating and modifying PDF documents in any JavaScript environment. + +#### Load and Manipulate Existing PDF +```javascript +import { PDFDocument } from 'pdf-lib'; +import fs from 'fs'; + +async function manipulatePDF() { + // Load existing PDF + const existingPdfBytes = fs.readFileSync('input.pdf'); + const pdfDoc = await PDFDocument.load(existingPdfBytes); + + // Get page count + const pageCount = pdfDoc.getPageCount(); + console.log(`Document has ${pageCount} pages`); + + // Add new page + const newPage = pdfDoc.addPage([600, 400]); + newPage.drawText('Added by pdf-lib', { + x: 100, + y: 300, + size: 16 + }); + + // Save modified PDF + const pdfBytes = await pdfDoc.save(); + fs.writeFileSync('modified.pdf', pdfBytes); +} +``` + +#### Create Complex PDFs from Scratch +```javascript +import { PDFDocument, rgb, StandardFonts } from 'pdf-lib'; +import fs from 'fs'; + +async function createPDF() { + const pdfDoc = await PDFDocument.create(); + + // Add fonts + const helveticaFont = await pdfDoc.embedFont(StandardFonts.Helvetica); + const helveticaBold = await pdfDoc.embedFont(StandardFonts.HelveticaBold); + + // Add page + const page = pdfDoc.addPage([595, 842]); // A4 size + const { width, height } = page.getSize(); + + // Add text with styling + page.drawText('Invoice #12345', { + x: 50, + y: height - 50, + size: 18, + font: helveticaBold, + color: rgb(0.2, 0.2, 0.8) + }); + + // Add rectangle (header background) + page.drawRectangle({ + x: 40, + y: height - 100, + width: width - 80, + height: 30, + color: rgb(0.9, 0.9, 0.9) + }); + + // Add table-like content + const items = [ + ['Item', 'Qty', 'Price', 'Total'], + ['Widget', '2', '$50', '$100'], + ['Gadget', '1', '$75', '$75'] + ]; + + let yPos = height - 150; + items.forEach(row => { + let xPos = 50; + row.forEach(cell => { + page.drawText(cell, { + x: xPos, + y: yPos, + size: 12, + font: helveticaFont + }); + xPos += 120; + }); + yPos -= 25; + }); + + const pdfBytes = await pdfDoc.save(); + fs.writeFileSync('created.pdf', pdfBytes); +} +``` + +#### Advanced Merge and Split Operations +```javascript +import { PDFDocument } from 'pdf-lib'; +import fs from 'fs'; + +async function mergePDFs() { + // Create new document + const mergedPdf = await PDFDocument.create(); + + // Load source PDFs + const pdf1Bytes = fs.readFileSync('doc1.pdf'); + const pdf2Bytes = fs.readFileSync('doc2.pdf'); + + const pdf1 = await PDFDocument.load(pdf1Bytes); + const pdf2 = await PDFDocument.load(pdf2Bytes); + + // Copy pages from first PDF + const pdf1Pages = await mergedPdf.copyPages(pdf1, pdf1.getPageIndices()); + pdf1Pages.forEach(page => mergedPdf.addPage(page)); + + // Copy specific pages from second PDF (pages 0, 2, 4) + const pdf2Pages = await mergedPdf.copyPages(pdf2, [0, 2, 4]); + pdf2Pages.forEach(page => mergedPdf.addPage(page)); + + const mergedPdfBytes = await mergedPdf.save(); + fs.writeFileSync('merged.pdf', mergedPdfBytes); +} +``` + +### pdfjs-dist (Apache License) + +PDF.js is Mozilla's JavaScript library for rendering PDFs in the browser. + +#### Basic PDF Loading and Rendering +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +// Configure worker (important for performance) +pdfjsLib.GlobalWorkerOptions.workerSrc = './pdf.worker.js'; + +async function renderPDF() { + // Load PDF + const loadingTask = pdfjsLib.getDocument('document.pdf'); + const pdf = await loadingTask.promise; + + console.log(`Loaded PDF with ${pdf.numPages} pages`); + + // Get first page + const page = await pdf.getPage(1); + const viewport = page.getViewport({ scale: 1.5 }); + + // Render to canvas + const canvas = document.createElement('canvas'); + const context = canvas.getContext('2d'); + canvas.height = viewport.height; + canvas.width = viewport.width; + + const renderContext = { + canvasContext: context, + viewport: viewport + }; + + await page.render(renderContext).promise; + document.body.appendChild(canvas); +} +``` + +#### Extract Text with Coordinates +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +async function extractText() { + const loadingTask = pdfjsLib.getDocument('document.pdf'); + const pdf = await loadingTask.promise; + + let fullText = ''; + + // Extract text from all pages + for (let i = 1; i <= pdf.numPages; i++) { + const page = await pdf.getPage(i); + const textContent = await page.getTextContent(); + + const pageText = textContent.items + .map(item => item.str) + .join(' '); + + fullText += `\n--- Page ${i} ---\n${pageText}`; + + // Get text with coordinates for advanced processing + const textWithCoords = textContent.items.map(item => ({ + text: item.str, + x: item.transform[4], + y: item.transform[5], + width: item.width, + height: item.height + })); + } + + console.log(fullText); + return fullText; +} +``` + +#### Extract Annotations and Forms +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +async function extractAnnotations() { + const loadingTask = pdfjsLib.getDocument('annotated.pdf'); + const pdf = await loadingTask.promise; + + for (let i = 1; i <= pdf.numPages; i++) { + const page = await pdf.getPage(i); + const annotations = await page.getAnnotations(); + + annotations.forEach(annotation => { + console.log(`Annotation type: ${annotation.subtype}`); + console.log(`Content: ${annotation.contents}`); + console.log(`Coordinates: ${JSON.stringify(annotation.rect)}`); + }); + } +} +``` + +## Advanced Command-Line Operations + +### poppler-utils Advanced Features + +#### Extract Text with Bounding Box Coordinates +```bash +# Extract text with bounding box coordinates (essential for structured data) +pdftotext -bbox-layout document.pdf output.xml + +# The XML output contains precise coordinates for each text element +``` + +#### Advanced Image Conversion +```bash +# Convert to PNG images with specific resolution +pdftoppm -png -r 300 document.pdf output_prefix + +# Convert specific page range with high resolution +pdftoppm -png -r 600 -f 1 -l 3 document.pdf high_res_pages + +# Convert to JPEG with quality setting +pdftoppm -jpeg -jpegopt quality=85 -r 200 document.pdf jpeg_output +``` + +#### Extract Embedded Images +```bash +# Extract all embedded images with metadata +pdfimages -j -p document.pdf page_images + +# List image info without extracting +pdfimages -list document.pdf + +# Extract images in their original format +pdfimages -all document.pdf images/img +``` + +### qpdf Advanced Features + +#### Complex Page Manipulation +```bash +# Split PDF into groups of pages +qpdf --split-pages=3 input.pdf output_group_%02d.pdf + +# Extract specific pages with complex ranges +qpdf input.pdf --pages input.pdf 1,3-5,8,10-end -- extracted.pdf + +# Merge specific pages from multiple PDFs +qpdf --empty --pages doc1.pdf 1-3 doc2.pdf 5-7 doc3.pdf 2,4 -- combined.pdf +``` + +#### PDF Optimization and Repair +```bash +# Optimize PDF for web (linearize for streaming) +qpdf --linearize input.pdf optimized.pdf + +# Remove unused objects and compress +qpdf --optimize-level=all input.pdf compressed.pdf + +# Attempt to repair corrupted PDF structure +qpdf --check input.pdf +qpdf --fix-qdf damaged.pdf repaired.pdf + +# Show detailed PDF structure for debugging +qpdf --show-all-pages input.pdf > structure.txt +``` + +#### Advanced Encryption +```bash +# Add password protection with specific permissions +qpdf --encrypt user_pass owner_pass 256 --print=none --modify=none -- input.pdf encrypted.pdf + +# Check encryption status +qpdf --show-encryption encrypted.pdf + +# Remove password protection (requires password) +qpdf --password=secret123 --decrypt encrypted.pdf decrypted.pdf +``` + +## Advanced Python Techniques + +### pdfplumber Advanced Features + +#### Extract Text with Precise Coordinates +```python +import pdfplumber + +with pdfplumber.open("document.pdf") as pdf: + page = pdf.pages[0] + + # Extract all text with coordinates + chars = page.chars + for char in chars[:10]: # First 10 characters + print(f"Char: '{char['text']}' at x:{char['x0']:.1f} y:{char['y0']:.1f}") + + # Extract text by bounding box (left, top, right, bottom) + bbox_text = page.within_bbox((100, 100, 400, 200)).extract_text() +``` + +#### Advanced Table Extraction with Custom Settings +```python +import pdfplumber +import pandas as pd + +with pdfplumber.open("complex_table.pdf") as pdf: + page = pdf.pages[0] + + # Extract tables with custom settings for complex layouts + table_settings = { + "vertical_strategy": "lines", + "horizontal_strategy": "lines", + "snap_tolerance": 3, + "intersection_tolerance": 15 + } + tables = page.extract_tables(table_settings) + + # Visual debugging for table extraction + img = page.to_image(resolution=150) + img.save("debug_layout.png") +``` + +### reportlab Advanced Features + +#### Create Professional Reports with Tables +```python +from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph +from reportlab.lib.styles import getSampleStyleSheet +from reportlab.lib import colors + +# Sample data +data = [ + ['Product', 'Q1', 'Q2', 'Q3', 'Q4'], + ['Widgets', '120', '135', '142', '158'], + ['Gadgets', '85', '92', '98', '105'] +] + +# Create PDF with table +doc = SimpleDocTemplate("report.pdf") +elements = [] + +# Add title +styles = getSampleStyleSheet() +title = Paragraph("Quarterly Sales Report", styles['Title']) +elements.append(title) + +# Add table with advanced styling +table = Table(data) +table.setStyle(TableStyle([ + ('BACKGROUND', (0, 0), (-1, 0), colors.grey), + ('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke), + ('ALIGN', (0, 0), (-1, -1), 'CENTER'), + ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'), + ('FONTSIZE', (0, 0), (-1, 0), 14), + ('BOTTOMPADDING', (0, 0), (-1, 0), 12), + ('BACKGROUND', (0, 1), (-1, -1), colors.beige), + ('GRID', (0, 0), (-1, -1), 1, colors.black) +])) +elements.append(table) + +doc.build(elements) +``` + +## Complex Workflows + +### Extract Figures/Images from PDF + +#### Method 1: Using pdfimages (fastest) +```bash +# Extract all images with original quality +pdfimages -all document.pdf images/img +``` + +#### Method 2: Using pypdfium2 + Image Processing +```python +import pypdfium2 as pdfium +from PIL import Image +import numpy as np + +def extract_figures(pdf_path, output_dir): + pdf = pdfium.PdfDocument(pdf_path) + + for page_num, page in enumerate(pdf): + # Render high-resolution page + bitmap = page.render(scale=3.0) + img = bitmap.to_pil() + + # Convert to numpy for processing + img_array = np.array(img) + + # Simple figure detection (non-white regions) + mask = np.any(img_array != [255, 255, 255], axis=2) + + # Find contours and extract bounding boxes + # (This is simplified - real implementation would need more sophisticated detection) + + # Save detected figures + # ... implementation depends on specific needs +``` + +### Batch PDF Processing with Error Handling +```python +import os +import glob +from pypdf import PdfReader, PdfWriter +import logging + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +def batch_process_pdfs(input_dir, operation='merge'): + pdf_files = glob.glob(os.path.join(input_dir, "*.pdf")) + + if operation == 'merge': + writer = PdfWriter() + for pdf_file in pdf_files: + try: + reader = PdfReader(pdf_file) + for page in reader.pages: + writer.add_page(page) + logger.info(f"Processed: {pdf_file}") + except Exception as e: + logger.error(f"Failed to process {pdf_file}: {e}") + continue + + with open("batch_merged.pdf", "wb") as output: + writer.write(output) + + elif operation == 'extract_text': + for pdf_file in pdf_files: + try: + reader = PdfReader(pdf_file) + text = "" + for page in reader.pages: + text += page.extract_text() + + output_file = pdf_file.replace('.pdf', '.txt') + with open(output_file, 'w', encoding='utf-8') as f: + f.write(text) + logger.info(f"Extracted text from: {pdf_file}") + + except Exception as e: + logger.error(f"Failed to extract text from {pdf_file}: {e}") + continue +``` + +### Advanced PDF Cropping +```python +from pypdf import PdfWriter, PdfReader + +reader = PdfReader("input.pdf") +writer = PdfWriter() + +# Crop page (left, bottom, right, top in points) +page = reader.pages[0] +page.mediabox.left = 50 +page.mediabox.bottom = 50 +page.mediabox.right = 550 +page.mediabox.top = 750 + +writer.add_page(page) +with open("cropped.pdf", "wb") as output: + writer.write(output) +``` + +## Performance Optimization Tips + +### 1. For Large PDFs +- Use streaming approaches instead of loading entire PDF in memory +- Use `qpdf --split-pages` for splitting large files +- Process pages individually with pypdfium2 + +### 2. For Text Extraction +- `pdftotext -bbox-layout` is fastest for plain text extraction +- Use pdfplumber for structured data and tables +- Avoid `pypdf.extract_text()` for very large documents + +### 3. For Image Extraction +- `pdfimages` is much faster than rendering pages +- Use low resolution for previews, high resolution for final output + +### 4. For Form Filling +- pdf-lib maintains form structure better than most alternatives +- Pre-validate form fields before processing + +### 5. Memory Management +```python +# Process PDFs in chunks +def process_large_pdf(pdf_path, chunk_size=10): + reader = PdfReader(pdf_path) + total_pages = len(reader.pages) + + for start_idx in range(0, total_pages, chunk_size): + end_idx = min(start_idx + chunk_size, total_pages) + writer = PdfWriter() + + for i in range(start_idx, end_idx): + writer.add_page(reader.pages[i]) + + # Process chunk + with open(f"chunk_{start_idx//chunk_size}.pdf", "wb") as output: + writer.write(output) +``` + +## Troubleshooting Common Issues + +### Encrypted PDFs +```python +# Handle password-protected PDFs +from pypdf import PdfReader + +try: + reader = PdfReader("encrypted.pdf") + if reader.is_encrypted: + reader.decrypt("password") +except Exception as e: + print(f"Failed to decrypt: {e}") +``` + +### Corrupted PDFs +```bash +# Use qpdf to repair +qpdf --check corrupted.pdf +qpdf --replace-input corrupted.pdf +``` + +### Text Extraction Issues +```python +# Fallback to OCR for scanned PDFs +import pytesseract +from pdf2image import convert_from_path + +def extract_text_with_ocr(pdf_path): + images = convert_from_path(pdf_path) + text = "" + for i, image in enumerate(images): + text += pytesseract.image_to_string(image) + return text +``` + +## License Information + +- **pypdf**: BSD License +- **pdfplumber**: MIT License +- **pypdfium2**: Apache/BSD License +- **reportlab**: BSD License +- **poppler-utils**: GPL-2 License +- **qpdf**: Apache License +- **pdf-lib**: MIT License +- **pdfjs-dist**: Apache License \ No newline at end of file diff --git a/skills/pptx/SKILL.md b/skills/pptx/SKILL.md new file mode 100644 index 00000000..a5da3017 --- /dev/null +++ b/skills/pptx/SKILL.md @@ -0,0 +1,263 @@ +--- +name: PowerPoint Presentation +description: "Use this skill any time a .pptx file is involved in any way -- as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx file (even if the extracted content will be used elsewhere, like in an email or summary); editing, modifying, or updating existing presentations; combining or splitting slide files; working with templates, layouts, speaker notes, or comments. Trigger whenever the user mentions \"deck,\" \"slides,\" \"presentation,\" or references a .pptx filename, regardless of what they plan to do with the content afterward. If a .pptx file needs to be opened, created, or touched, use this skill." +version: 1.0.0 +metadata: + emoji: "πŸ“Š" + tags: + - office + - presentation + - pptx + install: + - id: brew-libreoffice + kind: brew + formula: libreoffice + bins: [soffice] + label: "Install LibreOffice for PDF conversion" + os: [darwin] + - id: brew-poppler + kind: brew + formula: poppler + bins: [pdftoppm] + label: "Install poppler for PDF to image conversion" + os: [darwin, linux] + - id: npm-pptxgenjs + kind: node + formula: pptxgenjs + bins: [] + label: "Install PptxGenJS for creating presentations" + - id: npm-markitdown + kind: uv + formula: "markitdown[pptx]" + bins: [markitdown] + label: "Install markitdown for text extraction" +userInvocable: true +disableModelInvocation: false +--- + +# PPTX Skill + +## Quick Reference + +| Task | Guide | +|------|-------| +| Read/analyze content | `python -m markitdown presentation.pptx` | +| Edit or create from template | Read [editing.md](editing.md) | +| Create from scratch | Read [pptxgenjs.md](pptxgenjs.md) | + +--- + +## Reading Content + +```bash +# Text extraction +python -m markitdown presentation.pptx + +# Visual overview +python scripts/thumbnail.py presentation.pptx + +# Raw XML +python scripts/office/unpack.py presentation.pptx unpacked/ +``` + +--- + +## Editing Workflow + +**Read [editing.md](editing.md) for full details.** + +1. Analyze template with `thumbnail.py` +2. Unpack β†’ manipulate slides β†’ edit content β†’ clean β†’ pack + +--- + +## Creating from Scratch + +**Read [pptxgenjs.md](pptxgenjs.md) for full details.** + +Use when no template or reference presentation is available. + +--- + +## Design Ideas + +**Don't create boring slides.** Plain bullets on a white background won't impress anyone. Consider ideas from this list for each slide. + +### Before Starting + +- **Pick a bold, content-informed color palette**: The palette should feel designed for THIS topic. If swapping your colors into a completely different presentation would still "work," you haven't made specific enough choices. +- **Dominance over equality**: One color should dominate (60-70% visual weight), with 1-2 supporting tones and one sharp accent. Never give all colors equal weight. +- **Dark/light contrast**: Dark backgrounds for title + conclusion slides, light for content ("sandwich" structure). Or commit to dark throughout for a premium feel. +- **Commit to a visual motif**: Pick ONE distinctive element and repeat it -- rounded image frames, icons in colored circles, thick single-side borders. Carry it across every slide. + +### Color Palettes + +Choose colors that match your topic -- don't default to generic blue. Use these palettes as inspiration: + +| Theme | Primary | Secondary | Accent | +|-------|---------|-----------|--------| +| **Midnight Executive** | `1E2761` (navy) | `CADCFC` (ice blue) | `FFFFFF` (white) | +| **Forest & Moss** | `2C5F2D` (forest) | `97BC62` (moss) | `F5F5F5` (cream) | +| **Coral Energy** | `F96167` (coral) | `F9E795` (gold) | `2F3C7E` (navy) | +| **Warm Terracotta** | `B85042` (terracotta) | `E7E8D1` (sand) | `A7BEAE` (sage) | +| **Ocean Gradient** | `065A82` (deep blue) | `1C7293` (teal) | `21295C` (midnight) | +| **Charcoal Minimal** | `36454F` (charcoal) | `F2F2F2` (off-white) | `212121` (black) | +| **Teal Trust** | `028090` (teal) | `00A896` (seafoam) | `02C39A` (mint) | +| **Berry & Cream** | `6D2E46` (berry) | `A26769` (dusty rose) | `ECE2D0` (cream) | +| **Sage Calm** | `84B59F` (sage) | `69A297` (eucalyptus) | `50808E` (slate) | +| **Cherry Bold** | `990011` (cherry) | `FCF6F5` (off-white) | `2F3C7E` (navy) | + +### For Each Slide + +**Every slide needs a visual element** -- image, chart, icon, or shape. Text-only slides are forgettable. + +**Layout options:** +- Two-column (text left, illustration on right) +- Icon + text rows (icon in colored circle, bold header, description below) +- 2x2 or 2x3 grid (image on one side, grid of content blocks on other) +- Half-bleed image (full left or right side) with content overlay + +**Data display:** +- Large stat callouts (big numbers 60-72pt with small labels below) +- Comparison columns (before/after, pros/cons, side-by-side options) +- Timeline or process flow (numbered steps, arrows) + +**Visual polish:** +- Icons in small colored circles next to section headers +- Italic accent text for key stats or taglines + +### Typography + +**Choose an interesting font pairing** -- don't default to Arial. Pick a header font with personality and pair it with a clean body font. + +| Header Font | Body Font | +|-------------|-----------| +| Georgia | Calibri | +| Arial Black | Arial | +| Calibri | Calibri Light | +| Cambria | Calibri | +| Trebuchet MS | Calibri | +| Impact | Arial | +| Palatino | Garamond | +| Consolas | Calibri | + +| Element | Size | +|---------|------| +| Slide title | 36-44pt bold | +| Section header | 20-24pt bold | +| Body text | 14-16pt | +| Captions | 10-12pt muted | + +### Spacing + +- 0.5" minimum margins +- 0.3-0.5" between content blocks +- Leave breathing room--don't fill every inch + +### Avoid (Common Mistakes) + +- **Don't repeat the same layout** -- vary columns, cards, and callouts across slides +- **Don't center body text** -- left-align paragraphs and lists; center only titles +- **Don't skimp on size contrast** -- titles need 36pt+ to stand out from 14-16pt body +- **Don't default to blue** -- pick colors that reflect the specific topic +- **Don't mix spacing randomly** -- choose 0.3" or 0.5" gaps and use consistently +- **Don't style one slide and leave the rest plain** -- commit fully or keep it simple throughout +- **Don't create text-only slides** -- add images, icons, charts, or visual elements; avoid plain title + bullets +- **Don't forget text box padding** -- when aligning lines or shapes with text edges, set `margin: 0` on the text box or offset the shape to account for padding +- **Don't use low-contrast elements** -- icons AND text need strong contrast against the background; avoid light text on light backgrounds or dark text on dark backgrounds +- **NEVER use accent lines under titles** -- these are a hallmark of AI-generated slides; use whitespace or background color instead + +--- + +## QA (Required) + +**Assume there are problems. Your job is to find them.** + +Your first render is almost never correct. Approach QA as a bug hunt, not a confirmation step. If you found zero issues on first inspection, you weren't looking hard enough. + +### Content QA + +```bash +python -m markitdown output.pptx +``` + +Check for missing content, typos, wrong order. + +**When using templates, check for leftover placeholder text:** + +```bash +python -m markitdown output.pptx | grep -iE "xxxx|lorem|ipsum|this.*(page|slide).*layout" +``` + +If grep returns results, fix them before declaring success. + +### Visual QA + +**USE SUBAGENTS** -- even for 2-3 slides. You've been staring at the code and will see what you expect, not what's there. Subagents have fresh eyes. + +Convert slides to images (see [Converting to Images](#converting-to-images)), then use this prompt: + +``` +Visually inspect these slides. Assume there are issues -- find them. + +Look for: +- Overlapping elements (text through shapes, lines through words, stacked elements) +- Text overflow or cut off at edges/box boundaries +- Decorative lines positioned for single-line text but title wrapped to two lines +- Source citations or footers colliding with content above +- Elements too close (< 0.3" gaps) or cards/sections nearly touching +- Uneven gaps (large empty area in one place, cramped in another) +- Insufficient margin from slide edges (< 0.5") +- Columns or similar elements not aligned consistently +- Low-contrast text (e.g., light gray text on cream-colored background) +- Low-contrast icons (e.g., dark icons on dark backgrounds without a contrasting circle) +- Text boxes too narrow causing excessive wrapping +- Leftover placeholder content + +For each slide, list issues or areas of concern, even if minor. + +Read and analyze these images: +1. /path/to/slide-01.jpg (Expected: [brief description]) +2. /path/to/slide-02.jpg (Expected: [brief description]) + +Report ALL issues found, including minor ones. +``` + +### Verification Loop + +1. Generate slides β†’ Convert to images β†’ Inspect +2. **List issues found** (if none found, look again more critically) +3. Fix issues +4. **Re-verify affected slides** -- one fix often creates another problem +5. Repeat until a full pass reveals no new issues + +**Do not declare success until you've completed at least one fix-and-verify cycle.** + +--- + +## Converting to Images + +Convert presentations to individual slide images for visual inspection: + +```bash +python scripts/office/soffice.py --headless --convert-to pdf output.pptx +pdftoppm -jpeg -r 150 output.pdf slide +``` + +This creates `slide-01.jpg`, `slide-02.jpg`, etc. + +To re-render specific slides after fixes: + +```bash +pdftoppm -jpeg -r 150 -f N -l N output.pdf slide-fixed +``` + +--- + +## Dependencies + +- `pip install "markitdown[pptx]"` - text extraction +- `pip install Pillow` - thumbnail grids +- `npm install -g pptxgenjs` - creating from scratch +- LibreOffice (`soffice`) - PDF conversion (auto-configured for sandboxed environments via `scripts/office/soffice.py`) +- Poppler (`pdftoppm`) - PDF to images diff --git a/skills/pptx/editing.md b/skills/pptx/editing.md new file mode 100644 index 00000000..f873e8a0 --- /dev/null +++ b/skills/pptx/editing.md @@ -0,0 +1,205 @@ +# Editing Presentations + +## Template-Based Workflow + +When using an existing presentation as a template: + +1. **Analyze existing slides**: + ```bash + python scripts/thumbnail.py template.pptx + python -m markitdown template.pptx + ``` + Review `thumbnails.jpg` to see layouts, and markitdown output to see placeholder text. + +2. **Plan slide mapping**: For each content section, choose a template slide. + + ⚠️ **USE VARIED LAYOUTS** β€” monotonous presentations are a common failure mode. Don't default to basic title + bullet slides. Actively seek out: + - Multi-column layouts (2-column, 3-column) + - Image + text combinations + - Full-bleed images with text overlay + - Quote or callout slides + - Section dividers + - Stat/number callouts + - Icon grids or icon + text rows + + **Avoid:** Repeating the same text-heavy layout for every slide. + + Match content type to layout style (e.g., key points β†’ bullet slide, team info β†’ multi-column, testimonials β†’ quote slide). + +3. **Unpack**: `python scripts/office/unpack.py template.pptx unpacked/` + +4. **Build presentation** (do this yourself, not with subagents): + - Delete unwanted slides (remove from ``) + - Duplicate slides you want to reuse (`add_slide.py`) + - Reorder slides in `` + - **Complete all structural changes before step 5** + +5. **Edit content**: Update text in each `slide{N}.xml`. + **Use subagents here if available** β€” slides are separate XML files, so subagents can edit in parallel. + +6. **Clean**: `python scripts/clean.py unpacked/` + +7. **Pack**: `python scripts/office/pack.py unpacked/ output.pptx --original template.pptx` + +--- + +## Scripts + +| Script | Purpose | +|--------|---------| +| `unpack.py` | Extract and pretty-print PPTX | +| `add_slide.py` | Duplicate slide or create from layout | +| `clean.py` | Remove orphaned files | +| `pack.py` | Repack with validation | +| `thumbnail.py` | Create visual grid of slides | + +### unpack.py + +```bash +python scripts/office/unpack.py input.pptx unpacked/ +``` + +Extracts PPTX, pretty-prints XML, escapes smart quotes. + +### add_slide.py + +```bash +python scripts/add_slide.py unpacked/ slide2.xml # Duplicate slide +python scripts/add_slide.py unpacked/ slideLayout2.xml # From layout +``` + +Prints `` to add to `` at desired position. + +### clean.py + +```bash +python scripts/clean.py unpacked/ +``` + +Removes slides not in ``, unreferenced media, orphaned rels. + +### pack.py + +```bash +python scripts/office/pack.py unpacked/ output.pptx --original input.pptx +``` + +Validates, repairs, condenses XML, re-encodes smart quotes. + +### thumbnail.py + +```bash +python scripts/thumbnail.py input.pptx [output_prefix] [--cols N] +``` + +Creates `thumbnails.jpg` with slide filenames as labels. Default 3 columns, max 12 per grid. + +**Use for template analysis only** (choosing layouts). For visual QA, use `soffice` + `pdftoppm` to create full-resolution individual slide imagesβ€”see SKILL.md. + +--- + +## Slide Operations + +Slide order is in `ppt/presentation.xml` β†’ ``. + +**Reorder**: Rearrange `` elements. + +**Delete**: Remove ``, then run `clean.py`. + +**Add**: Use `add_slide.py`. Never manually copy slide filesβ€”the script handles notes references, Content_Types.xml, and relationship IDs that manual copying misses. + +--- + +## Editing Content + +**Subagents:** If available, use them here (after completing step 4). Each slide is a separate XML file, so subagents can edit in parallel. In your prompt to subagents, include: +- The slide file path(s) to edit +- **"Use the Edit tool for all changes"** +- The formatting rules and common pitfalls below + +For each slide: +1. Read the slide's XML +2. Identify ALL placeholder contentβ€”text, images, charts, icons, captions +3. Replace each placeholder with final content + +**Use the Edit tool, not sed or Python scripts.** The Edit tool forces specificity about what to replace and where, yielding better reliability. + +### Formatting Rules + +- **Bold all headers, subheadings, and inline labels**: Use `b="1"` on ``. This includes: + - Slide titles + - Section headers within a slide + - Inline labels like (e.g.: "Status:", "Description:") at the start of a line +- **Never use unicode bullets (β€’)**: Use proper list formatting with `` or `` +- **Bullet consistency**: Let bullets inherit from the layout. Only specify `` or ``. + +--- + +## Common Pitfalls + +### Template Adaptation + +When source content has fewer items than the template: +- **Remove excess elements entirely** (images, shapes, text boxes), don't just clear text +- Check for orphaned visuals after clearing text content +- Run visual QA to catch mismatched counts + +When replacing text with different length content: +- **Shorter replacements**: Usually safe +- **Longer replacements**: May overflow or wrap unexpectedly +- Test with visual QA after text changes +- Consider truncating or splitting content to fit the template's design constraints + +**Template slots β‰  Source items**: If template has 4 team members but source has 3 users, delete the 4th member's entire group (image + text boxes), not just the text. + +### Multi-Item Content + +If source has multiple items (numbered lists, multiple sections), create separate `` elements for each β€” **never concatenate into one string**. + +**❌ WRONG** β€” all items in one paragraph: +```xml + + Step 1: Do the first thing. Step 2: Do the second thing. + +``` + +**βœ… CORRECT** β€” separate paragraphs with bold headers: +```xml + + + Step 1 + + + + Do the first thing. + + + + Step 2 + + +``` + +Copy `` from the original paragraph to preserve line spacing. Use `b="1"` on headers. + +### Smart Quotes + +Handled automatically by unpack/pack. But the Edit tool converts smart quotes to ASCII. + +**When adding new text with quotes, use XML entities:** + +```xml +the “Agreement” +``` + +| Character | Name | Unicode | XML Entity | +|-----------|------|---------|------------| +| `β€œ` | Left double quote | U+201C | `“` | +| `”` | Right double quote | U+201D | `”` | +| `β€˜` | Left single quote | U+2018 | `‘` | +| `’` | Right single quote | U+2019 | `’` | + +### Other + +- **Whitespace**: Use `xml:space="preserve"` on `` with leading/trailing spaces +- **XML parsing**: Use `defusedxml.minidom`, not `xml.etree.ElementTree` (corrupts namespaces) diff --git a/skills/pptx/pptxgenjs.md b/skills/pptx/pptxgenjs.md new file mode 100644 index 00000000..6bfed908 --- /dev/null +++ b/skills/pptx/pptxgenjs.md @@ -0,0 +1,420 @@ +# PptxGenJS Tutorial + +## Setup & Basic Structure + +```javascript +const pptxgen = require("pptxgenjs"); + +let pres = new pptxgen(); +pres.layout = 'LAYOUT_16x9'; // or 'LAYOUT_16x10', 'LAYOUT_4x3', 'LAYOUT_WIDE' +pres.author = 'Your Name'; +pres.title = 'Presentation Title'; + +let slide = pres.addSlide(); +slide.addText("Hello World!", { x: 0.5, y: 0.5, fontSize: 36, color: "363636" }); + +pres.writeFile({ fileName: "Presentation.pptx" }); +``` + +## Layout Dimensions + +Slide dimensions (coordinates in inches): +- `LAYOUT_16x9`: 10" Γ— 5.625" (default) +- `LAYOUT_16x10`: 10" Γ— 6.25" +- `LAYOUT_4x3`: 10" Γ— 7.5" +- `LAYOUT_WIDE`: 13.3" Γ— 7.5" + +--- + +## Text & Formatting + +```javascript +// Basic text +slide.addText("Simple Text", { + x: 1, y: 1, w: 8, h: 2, fontSize: 24, fontFace: "Arial", + color: "363636", bold: true, align: "center", valign: "middle" +}); + +// Character spacing (use charSpacing, not letterSpacing which is silently ignored) +slide.addText("SPACED TEXT", { x: 1, y: 1, w: 8, h: 1, charSpacing: 6 }); + +// Rich text arrays +slide.addText([ + { text: "Bold ", options: { bold: true } }, + { text: "Italic ", options: { italic: true } } +], { x: 1, y: 3, w: 8, h: 1 }); + +// Multi-line text (requires breakLine: true) +slide.addText([ + { text: "Line 1", options: { breakLine: true } }, + { text: "Line 2", options: { breakLine: true } }, + { text: "Line 3" } // Last item doesn't need breakLine +], { x: 0.5, y: 0.5, w: 8, h: 2 }); + +// Text box margin (internal padding) +slide.addText("Title", { + x: 0.5, y: 0.3, w: 9, h: 0.6, + margin: 0 // Use 0 when aligning text with other elements like shapes or icons +}); +``` + +**Tip:** Text boxes have internal margin by default. Set `margin: 0` when you need text to align precisely with shapes, lines, or icons at the same x-position. + +--- + +## Lists & Bullets + +```javascript +// βœ… CORRECT: Multiple bullets +slide.addText([ + { text: "First item", options: { bullet: true, breakLine: true } }, + { text: "Second item", options: { bullet: true, breakLine: true } }, + { text: "Third item", options: { bullet: true } } +], { x: 0.5, y: 0.5, w: 8, h: 3 }); + +// ❌ WRONG: Never use unicode bullets +slide.addText("β€’ First item", { ... }); // Creates double bullets + +// Sub-items and numbered lists +{ text: "Sub-item", options: { bullet: true, indentLevel: 1 } } +{ text: "First", options: { bullet: { type: "number" }, breakLine: true } } +``` + +--- + +## Shapes + +```javascript +slide.addShape(pres.shapes.RECTANGLE, { + x: 0.5, y: 0.8, w: 1.5, h: 3.0, + fill: { color: "FF0000" }, line: { color: "000000", width: 2 } +}); + +slide.addShape(pres.shapes.OVAL, { x: 4, y: 1, w: 2, h: 2, fill: { color: "0000FF" } }); + +slide.addShape(pres.shapes.LINE, { + x: 1, y: 3, w: 5, h: 0, line: { color: "FF0000", width: 3, dashType: "dash" } +}); + +// With transparency +slide.addShape(pres.shapes.RECTANGLE, { + x: 1, y: 1, w: 3, h: 2, + fill: { color: "0088CC", transparency: 50 } +}); + +// Rounded rectangle (rectRadius only works with ROUNDED_RECTANGLE, not RECTANGLE) +// ⚠️ Don't pair with rectangular accent overlays β€” they won't cover rounded corners. Use RECTANGLE instead. +slide.addShape(pres.shapes.ROUNDED_RECTANGLE, { + x: 1, y: 1, w: 3, h: 2, + fill: { color: "FFFFFF" }, rectRadius: 0.1 +}); + +// With shadow +slide.addShape(pres.shapes.RECTANGLE, { + x: 1, y: 1, w: 3, h: 2, + fill: { color: "FFFFFF" }, + shadow: { type: "outer", color: "000000", blur: 6, offset: 2, angle: 135, opacity: 0.15 } +}); +``` + +Shadow options: + +| Property | Type | Range | Notes | +|----------|------|-------|-------| +| `type` | string | `"outer"`, `"inner"` | | +| `color` | string | 6-char hex (e.g. `"000000"`) | No `#` prefix, no 8-char hex β€” see Common Pitfalls | +| `blur` | number | 0-100 pt | | +| `offset` | number | 0-200 pt | **Must be non-negative** β€” negative values corrupt the file | +| `angle` | number | 0-359 degrees | Direction the shadow falls (135 = bottom-right, 270 = upward) | +| `opacity` | number | 0.0-1.0 | Use this for transparency, never encode in color string | + +To cast a shadow upward (e.g. on a footer bar), use `angle: 270` with a positive offset β€” do **not** use a negative offset. + +**Note**: Gradient fills are not natively supported. Use a gradient image as a background instead. + +--- + +## Images + +### Image Sources + +```javascript +// From file path +slide.addImage({ path: "images/chart.png", x: 1, y: 1, w: 5, h: 3 }); + +// From URL +slide.addImage({ path: "https://example.com/image.jpg", x: 1, y: 1, w: 5, h: 3 }); + +// From base64 (faster, no file I/O) +slide.addImage({ data: "image/png;base64,iVBORw0KGgo...", x: 1, y: 1, w: 5, h: 3 }); +``` + +### Image Options + +```javascript +slide.addImage({ + path: "image.png", + x: 1, y: 1, w: 5, h: 3, + rotate: 45, // 0-359 degrees + rounding: true, // Circular crop + transparency: 50, // 0-100 + flipH: true, // Horizontal flip + flipV: false, // Vertical flip + altText: "Description", // Accessibility + hyperlink: { url: "https://example.com" } +}); +``` + +### Image Sizing Modes + +```javascript +// Contain - fit inside, preserve ratio +{ sizing: { type: 'contain', w: 4, h: 3 } } + +// Cover - fill area, preserve ratio (may crop) +{ sizing: { type: 'cover', w: 4, h: 3 } } + +// Crop - cut specific portion +{ sizing: { type: 'crop', x: 0.5, y: 0.5, w: 2, h: 2 } } +``` + +### Calculate Dimensions (preserve aspect ratio) + +```javascript +const origWidth = 1978, origHeight = 923, maxHeight = 3.0; +const calcWidth = maxHeight * (origWidth / origHeight); +const centerX = (10 - calcWidth) / 2; + +slide.addImage({ path: "image.png", x: centerX, y: 1.2, w: calcWidth, h: maxHeight }); +``` + +### Supported Formats + +- **Standard**: PNG, JPG, GIF (animated GIFs work in Microsoft 365) +- **SVG**: Works in modern PowerPoint/Microsoft 365 + +--- + +## Icons + +Use react-icons to generate SVG icons, then rasterize to PNG for universal compatibility. + +### Setup + +```javascript +const React = require("react"); +const ReactDOMServer = require("react-dom/server"); +const sharp = require("sharp"); +const { FaCheckCircle, FaChartLine } = require("react-icons/fa"); + +function renderIconSvg(IconComponent, color = "#000000", size = 256) { + return ReactDOMServer.renderToStaticMarkup( + React.createElement(IconComponent, { color, size: String(size) }) + ); +} + +async function iconToBase64Png(IconComponent, color, size = 256) { + const svg = renderIconSvg(IconComponent, color, size); + const pngBuffer = await sharp(Buffer.from(svg)).png().toBuffer(); + return "image/png;base64," + pngBuffer.toString("base64"); +} +``` + +### Add Icon to Slide + +```javascript +const iconData = await iconToBase64Png(FaCheckCircle, "#4472C4", 256); + +slide.addImage({ + data: iconData, + x: 1, y: 1, w: 0.5, h: 0.5 // Size in inches +}); +``` + +**Note**: Use size 256 or higher for crisp icons. The size parameter controls the rasterization resolution, not the display size on the slide (which is set by `w` and `h` in inches). + +### Icon Libraries + +Install: `npm install -g react-icons react react-dom sharp` + +Popular icon sets in react-icons: +- `react-icons/fa` - Font Awesome +- `react-icons/md` - Material Design +- `react-icons/hi` - Heroicons +- `react-icons/bi` - Bootstrap Icons + +--- + +## Slide Backgrounds + +```javascript +// Solid color +slide.background = { color: "F1F1F1" }; + +// Color with transparency +slide.background = { color: "FF3399", transparency: 50 }; + +// Image from URL +slide.background = { path: "https://example.com/bg.jpg" }; + +// Image from base64 +slide.background = { data: "image/png;base64,iVBORw0KGgo..." }; +``` + +--- + +## Tables + +```javascript +slide.addTable([ + ["Header 1", "Header 2"], + ["Cell 1", "Cell 2"] +], { + x: 1, y: 1, w: 8, h: 2, + border: { pt: 1, color: "999999" }, fill: { color: "F1F1F1" } +}); + +// Advanced with merged cells +let tableData = [ + [{ text: "Header", options: { fill: { color: "6699CC" }, color: "FFFFFF", bold: true } }, "Cell"], + [{ text: "Merged", options: { colspan: 2 } }] +]; +slide.addTable(tableData, { x: 1, y: 3.5, w: 8, colW: [4, 4] }); +``` + +--- + +## Charts + +```javascript +// Bar chart +slide.addChart(pres.charts.BAR, [{ + name: "Sales", labels: ["Q1", "Q2", "Q3", "Q4"], values: [4500, 5500, 6200, 7100] +}], { + x: 0.5, y: 0.6, w: 6, h: 3, barDir: 'col', + showTitle: true, title: 'Quarterly Sales' +}); + +// Line chart +slide.addChart(pres.charts.LINE, [{ + name: "Temp", labels: ["Jan", "Feb", "Mar"], values: [32, 35, 42] +}], { x: 0.5, y: 4, w: 6, h: 3, lineSize: 3, lineSmooth: true }); + +// Pie chart +slide.addChart(pres.charts.PIE, [{ + name: "Share", labels: ["A", "B", "Other"], values: [35, 45, 20] +}], { x: 7, y: 1, w: 5, h: 4, showPercent: true }); +``` + +### Better-Looking Charts + +Default charts look dated. Apply these options for a modern, clean appearance: + +```javascript +slide.addChart(pres.charts.BAR, chartData, { + x: 0.5, y: 1, w: 9, h: 4, barDir: "col", + + // Custom colors (match your presentation palette) + chartColors: ["0D9488", "14B8A6", "5EEAD4"], + + // Clean background + chartArea: { fill: { color: "FFFFFF" }, roundedCorners: true }, + + // Muted axis labels + catAxisLabelColor: "64748B", + valAxisLabelColor: "64748B", + + // Subtle grid (value axis only) + valGridLine: { color: "E2E8F0", size: 0.5 }, + catGridLine: { style: "none" }, + + // Data labels on bars + showValue: true, + dataLabelPosition: "outEnd", + dataLabelColor: "1E293B", + + // Hide legend for single series + showLegend: false, +}); +``` + +**Key styling options:** +- `chartColors: [...]` - hex colors for series/segments +- `chartArea: { fill, border, roundedCorners }` - chart background +- `catGridLine/valGridLine: { color, style, size }` - grid lines (`style: "none"` to hide) +- `lineSmooth: true` - curved lines (line charts) +- `legendPos: "r"` - legend position: "b", "t", "l", "r", "tr" + +--- + +## Slide Masters + +```javascript +pres.defineSlideMaster({ + title: 'TITLE_SLIDE', background: { color: '283A5E' }, + objects: [{ + placeholder: { options: { name: 'title', type: 'title', x: 1, y: 2, w: 8, h: 2 } } + }] +}); + +let titleSlide = pres.addSlide({ masterName: "TITLE_SLIDE" }); +titleSlide.addText("My Title", { placeholder: "title" }); +``` + +--- + +## Common Pitfalls + +⚠️ These issues cause file corruption, visual bugs, or broken output. Avoid them. + +1. **NEVER use "#" with hex colors** - causes file corruption + ```javascript + color: "FF0000" // βœ… CORRECT + color: "#FF0000" // ❌ WRONG + ``` + +2. **NEVER encode opacity in hex color strings** - 8-char colors (e.g., `"00000020"`) corrupt the file. Use the `opacity` property instead. + ```javascript + shadow: { type: "outer", blur: 6, offset: 2, color: "00000020" } // ❌ CORRUPTS FILE + shadow: { type: "outer", blur: 6, offset: 2, color: "000000", opacity: 0.12 } // βœ… CORRECT + ``` + +3. **Use `bullet: true`** - NEVER unicode symbols like "β€’" (creates double bullets) + +4. **Use `breakLine: true`** between array items or text runs together + +5. **Avoid `lineSpacing` with bullets** - causes excessive gaps; use `paraSpaceAfter` instead + +6. **Each presentation needs fresh instance** - don't reuse `pptxgen()` objects + +7. **NEVER reuse option objects across calls** - PptxGenJS mutates objects in-place (e.g. converting shadow values to EMU). Sharing one object between multiple calls corrupts the second shape. + ```javascript + const shadow = { type: "outer", blur: 6, offset: 2, color: "000000", opacity: 0.15 }; + slide.addShape(pres.shapes.RECTANGLE, { shadow, ... }); // ❌ second call gets already-converted values + slide.addShape(pres.shapes.RECTANGLE, { shadow, ... }); + + const makeShadow = () => ({ type: "outer", blur: 6, offset: 2, color: "000000", opacity: 0.15 }); + slide.addShape(pres.shapes.RECTANGLE, { shadow: makeShadow(), ... }); // βœ… fresh object each time + slide.addShape(pres.shapes.RECTANGLE, { shadow: makeShadow(), ... }); + ``` + +8. **Don't use `ROUNDED_RECTANGLE` with accent borders** - rectangular overlay bars won't cover rounded corners. Use `RECTANGLE` instead. + ```javascript + // ❌ WRONG: Accent bar doesn't cover rounded corners + slide.addShape(pres.shapes.ROUNDED_RECTANGLE, { x: 1, y: 1, w: 3, h: 1.5, fill: { color: "FFFFFF" } }); + slide.addShape(pres.shapes.RECTANGLE, { x: 1, y: 1, w: 0.08, h: 1.5, fill: { color: "0891B2" } }); + + // βœ… CORRECT: Use RECTANGLE for clean alignment + slide.addShape(pres.shapes.RECTANGLE, { x: 1, y: 1, w: 3, h: 1.5, fill: { color: "FFFFFF" } }); + slide.addShape(pres.shapes.RECTANGLE, { x: 1, y: 1, w: 0.08, h: 1.5, fill: { color: "0891B2" } }); + ``` + +--- + +## Quick Reference + +- **Shapes**: RECTANGLE, OVAL, LINE, ROUNDED_RECTANGLE +- **Charts**: BAR, LINE, PIE, DOUGHNUT, SCATTER, BUBBLE, RADAR +- **Layouts**: LAYOUT_16x9 (10"Γ—5.625"), LAYOUT_16x10, LAYOUT_4x3, LAYOUT_WIDE +- **Alignment**: "left", "center", "right" +- **Chart data labels**: "outEnd", "inEnd", "center" diff --git a/skills/profile-setup/SKILL.md b/skills/profile-setup/SKILL.md new file mode 100644 index 00000000..d8162073 --- /dev/null +++ b/skills/profile-setup/SKILL.md @@ -0,0 +1,91 @@ +--- +name: Profile Setup +description: Interactive setup wizard to personalize your agent profile +version: 1.0.0 +metadata: + emoji: "πŸ§™" + tags: + - profile + - setup + - onboarding +--- + +## Instructions + +You are conducting an interactive setup to personalize the agent profile. Your goal is to learn about the user through natural conversation and update their profile files accordingly. + +### Setup Context + +The user has just created a new agent profile and wants to personalize it. You have access to the profile directory and can update the following files: + +- `soul.md` - Agent identity (name, role, style) +- `user.md` - Information about the user (name, preferences) + +### Conversation Flow + +Have a natural conversation to configure the agent and learn about the user. Don't follow a rigid script - adapt based on their responses. Here are topics to explore: + +1. **Agent Identity** (for soul.md) + - What would you like to call me? (agent's name) + - What personality/style do you prefer? (concise and direct, warm and friendly, formal, casual, etc.) + +2. **About the User** (for user.md) + - What should I call you? + - What's your timezone or location? (for context) + +3. **Communication Preferences** + - How do you prefer responses? (concise vs detailed) + - Any language preferences? (English, Chinese, mixed) + - Anything that annoys you in AI responses? + +### Guidelines + +- **Be conversational**: This is a dialogue, not an interrogation. Ask follow-up questions naturally. +- **Don't ask everything**: Pick the most relevant questions based on context. Skip what doesn't apply. +- **Summarize and confirm**: After gathering information, summarize what you learned and ask if it's accurate. +- **Update files progressively**: As you learn things, update the relevant profile files. +- **End gracefully**: When you have enough information, wrap up the conversation and let the user know their profile is ready. + +### File Updates + +When updating files, use the `edit` tool to modify specific sections: + +**soul.md - Update the Identity section:** +```markdown +## Identity + +- **Name:** Jarvis +- **Role:** General-purpose AI assistant +- **Style:** Concise, direct, and friendly +``` + +**user.md example:** +```markdown +# User + +- **Name:** Jiayuan +- **Call me:** Jiayuan +- **Timezone:** Asia/Shanghai + +## Preferences + +- Prefers concise responses +- Language: Chinese preferred, English for technical terms +``` + +### Starting the Conversation + +Begin with a friendly greeting and explain what you're doing. Start by asking about the agent's identity first, then move to learning about the user. For example: + +"Hi! I'm here to help set up your agent profile. Let me ask you a few questions so I can be configured to assist you better. + +First, what would you like to call me? (Or just press enter to keep the default name 'Assistant')" + +### Ending the Conversation + +When you've gathered enough information, summarize and close: + +"Great! I've updated your profile with what I learned: +- [Summary of key points] + +Your profile is ready. You can always update these files later or run setup again. Feel free to start chatting with me anytime!" diff --git a/skills/skill-creator/SKILL.md b/skills/skill-creator/SKILL.md new file mode 100644 index 00000000..4ba4cba1 --- /dev/null +++ b/skills/skill-creator/SKILL.md @@ -0,0 +1,301 @@ +--- +name: Skill Creator +description: Create, edit, and manage custom skills to extend agent capabilities. Also activates inactive skills by guiding users through API key setup. Use when the user asks to create a new skill, build a custom capability, extend the agent's functionality, or when an inactive skill matches the user's intent. +version: 1.2.0 +metadata: + emoji: "πŸ› οΈ" + always: true + tags: + - meta + - skills + - developer-tools +--- + +## Instructions + +You can create, edit, and manage skills to extend your own capabilities or help users build custom skills. You also activate inactive skills by guiding users through API key configuration. + +## Activating Inactive Skills + +When a user's request matches an **inactive skill** (listed under "Installed But Inactive Skills" in your system prompt), follow this flow: + +1. **Inform the user**: Tell them the skill exists but needs setup +2. **Explain what's missing**: Reference the diagnostic info (e.g., "The Gmail skill requires a GMAIL_API_KEY") +3. **Guide them to get the key**: Use `web_search` or `web_fetch` to find how to obtain the required API key, then provide clear step-by-step instructions to the user +4. **Accept the key in chat**: Ask the user to paste the API key directly in the conversation +5. **Write the `.env` file**: Use the `write` tool to create the skill's `.env` file: + ``` + ~/.super-multica/skills//.env + ``` + Content format: + ``` + # API key for + = + ``` +6. **Confirm activation**: The skill system auto-reloads on file changes. Tell the user the skill is now active and proceed with their original request. + +**IMPORTANT**: The user's API key is written to a local file only. Never log, echo, or transmit the key anywhere else. + +### Example (hypothetical β€” only act on skills that actually appear in your system prompt) + +Suppose the system prompt contains an inactive skill entry like: +``` +- **Stock Tracker** (`stock-tracker`): Track stock prices + - Missing environment variables: STOCK_API_KEY + - Fix: Set STOCK_API_KEY in ~/.super-multica/skills/stock-tracker/.env +``` + +Then the conversation would be: +``` +User: "What's AAPL trading at?" +Agent: *sees stock-tracker in inactive skills list* +Agent: *uses web_search to find how to get a Stock API key* +Agent: "I have a Stock Tracker skill but it needs a STOCK_API_KEY. Here's how to get one: ..." +User: "sk-abc123..." +Agent: *writes ~/.super-multica/skills/stock-tracker/.env* +Agent: "Done! Stock Tracker is active. Let me check AAPL for you..." +``` + +**CRITICAL**: Only reference skills that are actually listed in your system prompt under "Installed But Inactive Skills". Never assume a skill exists without seeing it there. + +## Creating New Skills When No Match Exists + +If the user asks for a capability that doesn't match any existing or inactive skill: + +1. **Suggest creating a new skill** if the capability is well-defined and repeatable +2. Briefly describe what the skill would do and ask for confirmation +3. Follow the **Skill Creation Process** below to create it +4. If the new skill needs API keys, guide the user through obtaining and configuring them + +## Skill Creation Process + +**ALWAYS follow these steps in order when creating a new skill:** + +1. Understand what the skill should do +2. Initialize the skill using `init_skill.py` +3. Edit the generated SKILL.md +4. Test the skill + +### Step 1: Understand the Skill + +Before creating, clarify: +- What functionality should the skill provide? +- When should it be triggered? +- Does it need helper scripts? + +### Step 2: Initialize the Skill + +**CRITICAL: Never create skills in the current working directory.** + +**Choose the correct directory based on context:** +- **If running under a profile**: Create in `~/.super-multica/agent-profiles//skills/` (profile-specific) +- **If no profile**: Create in `~/.super-multica/skills/` (global) + +```bash +# For profile-specific skill (when running under a profile): +mkdir -p ~/.super-multica/agent-profiles//skills/ + +# For global skill (when no profile is active): +mkdir -p ~/.super-multica/skills/ +``` + +Create SKILL.md with proper structure: + +```bash +# Replace with the appropriate path from above +cat > /SKILL.md << 'EOF' +--- +name: +description: +version: 1.0.0 +metadata: + emoji: "πŸ”§" + tags: + - custom +--- + +## Instructions + + +EOF + +# (Optional) Create scripts directory if needed +mkdir -p /scripts +``` + +**Example - Creating a translator skill (global):** +```bash +mkdir -p ~/.super-multica/skills/translator + +cat > ~/.super-multica/skills/translator/SKILL.md << 'EOF' +--- +name: Translator +description: Translate text between languages. Use when user asks to translate text. +version: 1.0.0 +metadata: + emoji: "🌐" + tags: + - language +--- + +## Instructions + +When asked to translate text: +1. Identify source and target languages +2. Provide accurate, natural translations +3. For ambiguous terms, offer alternatives +EOF +``` + +### Step 3: Edit the Skill + +After initialization, edit the `SKILL.md` file in the skill directory: + +1. Update the `description` - This is the primary trigger mechanism +2. Write clear `## Instructions` - What the agent should do +3. Add helper scripts to `scripts/` if needed +4. Add reference docs to `references/` if needed + +### Step 4: Test the Skill + +The skill is automatically loaded (hot-reload). Verify with: +```bash +pnpm skills:cli list | grep +``` + +**IMPORTANT: Do NOT create .skill package files.** Skills are loaded directly from the directory structure. There is no packaging step needed. + +## SKILL.md Format + +Every skill must have a `SKILL.md` file with YAML frontmatter: + +```markdown +--- +name: Skill Display Name +description: Brief description of what this skill does +version: 1.0.0 +metadata: + emoji: "πŸ”§" + tags: + - category1 + requires: + bins: [required-binary] + env: [REQUIRED_ENV_VAR] +--- + +## Instructions + +Detailed instructions for using this skill... +``` + +### Frontmatter Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | Yes | Display name for the skill | +| `description` | Yes | Short description (triggers skill selection) | +| `version` | No | Semantic version | +| `metadata.emoji` | No | Emoji for display | +| `metadata.tags` | No | Categorization tags | +| `metadata.requires.bins` | No | Required binaries (all must exist) | +| `metadata.requires.anyBins` | No | Alternative binaries (one must exist) | +| `metadata.requires.env` | No | Required environment variables | + +## Directory Structure + +Skills are stored in two locations: + +``` +# Global skills (available to all profiles) +~/.super-multica/skills/ +β”œβ”€β”€ my-skill/ +β”‚ └── SKILL.md +└── another-skill/ + β”œβ”€β”€ SKILL.md + β”œβ”€β”€ scripts/ + β”‚ └── helper.py + └── references/ + └── api-docs.md + +# Profile-specific skills (only for this profile) +~/.super-multica/agent-profiles//skills/ +└── profile-only-skill/ + └── SKILL.md +``` + +## Editing Existing Skills + +To modify an existing skill: + +1. Read the current SKILL.md file +2. Make changes to frontmatter or instructions +3. Save - changes take effect immediately (hot-reload) + +## Listing and Removing Skills + +```bash +# List all skills +pnpm skills:cli list + +# Check skill status +pnpm skills:cli status + +# Remove a global skill +pnpm skills:cli remove +# or +rm -rf ~/.super-multica/skills/ + +# Remove a profile-specific skill +rm -rf ~/.super-multica/agent-profiles//skills/ +``` + +## Skills with API Key Requirements + +When creating a skill that needs an API key: + +1. Declare env requirements in the SKILL.md frontmatter: + ```yaml + metadata: + requires: + env: [SERVICE_API_KEY] + ``` + +2. After creating the SKILL.md, write the `.env` file in the same directory: + ``` + # API key for + SERVICE_API_KEY= + ``` + +3. The skill becomes eligible immediately (hot-reload is automatic). + +### .env File Format + +Each skill stores its credentials in `~/.super-multica/skills//.env`: + +``` +# Lines starting with # are comments +KEY_NAME=value +ANOTHER_KEY="value with spaces" +``` + +Rules: +- One key per line, `KEY=VALUE` format +- Quotes are optional (stripped automatically) +- Each skill has its own `.env` β€” no centralized credential file + +## Best Practices + +1. **Correct directory** - Never create skills in the current working directory +2. **Clear description** - Include "when to use" triggers in the description +3. **Concise instructions** - Keep SKILL.md under 500 lines +4. **Test scripts** - Run helper scripts to verify they work +5. **Single responsibility** - Each skill should do one thing well +6. **Proactive activation** - When you see an inactive skill matching user intent, suggest activating it + +## Skill Precedence + +Skills load from two sources (highest priority wins): +1. Profile-specific skills (`~/.super-multica/agent-profiles//skills/`) +2. Global skills (`~/.super-multica/skills/`) + +Profile skills override global skills with the same ID. diff --git a/skills/whisper/SKILL.md b/skills/whisper/SKILL.md new file mode 100644 index 00000000..fa9a128d --- /dev/null +++ b/skills/whisper/SKILL.md @@ -0,0 +1,55 @@ +--- +name: Audio Transcription +description: Transcribe audio files using local Whisper CLI when automatic pre-processing is unavailable +version: 1.1.0 +metadata: + emoji: "πŸŽ™οΈ" + requires: + anyBins: + - whisper + - whisper-cli + install: + - id: brew-whisper + kind: brew + formula: openai-whisper + bins: [whisper] + label: "Install OpenAI Whisper via Homebrew" + os: [darwin] + tags: + - audio + - transcription + - media +userInvocable: false +disableModelInvocation: false +--- + +## Audio Transcription (Agent Fallback) + +Voice messages from channels are pre-processed before reaching you. The transcription +priority is: + +1. **Local whisper CLI** (free, offline) β€” requires `whisper` or `whisper-cli` in PATH +2. **OpenAI Whisper API** β€” requires an OpenAI API key in credentials +3. **No provider available** β€” you receive a raw file path instead of a transcript + +When both providers are unavailable, you will receive `[audio message received]` with a +`File:` path instead of `[Voice Message]` with a transcript. Use local whisper to +transcribe manually: + +``` +whisper "" --model base --output_format txt --output_dir /tmp +``` + +Then read the `.txt` file from `/tmp/` and respond based on the transcribed content. + +### Setup + +To enable automatic local transcription (recommended): + +```bash +brew install openai-whisper +``` + +The first run will download the `base` model (~139MB) to `~/.cache/whisper/`. +No app restart is required β€” the binary is detected automatically on the next +voice message. diff --git a/skills/xlsx/SKILL.md b/skills/xlsx/SKILL.md new file mode 100644 index 00000000..dfdb1f6a --- /dev/null +++ b/skills/xlsx/SKILL.md @@ -0,0 +1,307 @@ +--- +name: Excel Spreadsheet +description: "Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved." +version: 1.0.0 +metadata: + emoji: "πŸ“—" + tags: + - office + - spreadsheet + - xlsx + install: + - id: brew-libreoffice + kind: brew + formula: libreoffice + bins: [soffice] + label: "Install LibreOffice for formula recalculation" + os: [darwin] +userInvocable: true +disableModelInvocation: false +--- + +# Requirements for Outputs + +## All Excel files + +### Professional Font +- Use a consistent, professional font (e.g., Arial, Times New Roman) for all deliverables unless otherwise instructed by the user + +### Zero Formula Errors +- Every Excel model MUST be delivered with ZERO formula errors (#REF!, #DIV/0!, #VALUE!, #N/A, #NAME?) + +### Preserve Existing Templates (when updating templates) +- Study and EXACTLY match existing format, style, and conventions when modifying files +- Never impose standardized formatting on files with established patterns +- Existing template conventions ALWAYS override these guidelines + +## Financial models + +### Color Coding Standards +Unless otherwise stated by the user or existing template + +#### Industry-Standard Color Conventions +- **Blue text (RGB: 0,0,255)**: Hardcoded inputs, and numbers users will change for scenarios +- **Black text (RGB: 0,0,0)**: ALL formulas and calculations +- **Green text (RGB: 0,128,0)**: Links pulling from other worksheets within same workbook +- **Red text (RGB: 255,0,0)**: External links to other files +- **Yellow background (RGB: 255,255,0)**: Key assumptions needing attention or cells that need to be updated + +### Number Formatting Standards + +#### Required Format Rules +- **Years**: Format as text strings (e.g., "2024" not "2,024") +- **Currency**: Use $#,##0 format; ALWAYS specify units in headers ("Revenue ($mm)") +- **Zeros**: Use number formatting to make all zeros "-", including percentages (e.g., "$#,##0;($#,##0);-") +- **Percentages**: Default to 0.0% format (one decimal) +- **Multiples**: Format as 0.0x for valuation multiples (EV/EBITDA, P/E) +- **Negative numbers**: Use parentheses (123) not minus -123 + +### Formula Construction Rules + +#### Assumptions Placement +- Place ALL assumptions (growth rates, margins, multiples, etc.) in separate assumption cells +- Use cell references instead of hardcoded values in formulas +- Example: Use =B5*(1+$B$6) instead of =B5*1.05 + +#### Formula Error Prevention +- Verify all cell references are correct +- Check for off-by-one errors in ranges +- Ensure consistent formulas across all projection periods +- Test with edge cases (zero values, negative numbers) +- Verify no unintended circular references + +#### Documentation Requirements for Hardcodes +- Comment or in cells beside (if end of table). Format: "Source: [System/Document], [Date], [Specific Reference], [URL if applicable]" +- Examples: + - "Source: Company 10-K, FY2024, Page 45, Revenue Note, [SEC EDGAR URL]" + - "Source: Company 10-Q, Q2 2025, Exhibit 99.1, [SEC EDGAR URL]" + - "Source: Bloomberg Terminal, 8/15/2025, AAPL US Equity" + - "Source: FactSet, 8/20/2025, Consensus Estimates Screen" + +# XLSX creation, editing, and analysis + +## Overview + +A user may ask you to create, edit, or analyze the contents of an .xlsx file. You have different tools and workflows available for different tasks. + +## Important Requirements + +**LibreOffice Required for Formula Recalculation**: You can assume LibreOffice is installed for recalculating formula values using the `scripts/recalc.py` script. The script automatically configures LibreOffice on first run, including in sandboxed environments where Unix sockets are restricted (handled by `scripts/office/soffice.py`) + +## Reading and analyzing data + +### Data analysis with pandas +For data analysis, visualization, and basic operations, use **pandas** which provides powerful data manipulation capabilities: + +```python +import pandas as pd + +# Read Excel +df = pd.read_excel('file.xlsx') # Default: first sheet +all_sheets = pd.read_excel('file.xlsx', sheet_name=None) # All sheets as dict + +# Analyze +df.head() # Preview data +df.info() # Column info +df.describe() # Statistics + +# Write Excel +df.to_excel('output.xlsx', index=False) +``` + +## Excel File Workflows + +## CRITICAL: Use Formulas, Not Hardcoded Values + +**Always use Excel formulas instead of calculating values in Python and hardcoding them.** This ensures the spreadsheet remains dynamic and updateable. + +### WRONG - Hardcoding Calculated Values +```python +# Bad: Calculating in Python and hardcoding result +total = df['Sales'].sum() +sheet['B10'] = total # Hardcodes 5000 + +# Bad: Computing growth rate in Python +growth = (df.iloc[-1]['Revenue'] - df.iloc[0]['Revenue']) / df.iloc[0]['Revenue'] +sheet['C5'] = growth # Hardcodes 0.15 + +# Bad: Python calculation for average +avg = sum(values) / len(values) +sheet['D20'] = avg # Hardcodes 42.5 +``` + +### CORRECT - Using Excel Formulas +```python +# Good: Let Excel calculate the sum +sheet['B10'] = '=SUM(B2:B9)' + +# Good: Growth rate as Excel formula +sheet['C5'] = '=(C4-C2)/C2' + +# Good: Average using Excel function +sheet['D20'] = '=AVERAGE(D2:D19)' +``` + +This applies to ALL calculations - totals, percentages, ratios, differences, etc. The spreadsheet should be able to recalculate when source data changes. + +## Common Workflow +1. **Choose tool**: pandas for data, openpyxl for formulas/formatting +2. **Create/Load**: Create new workbook or load existing file +3. **Modify**: Add/edit data, formulas, and formatting +4. **Save**: Write to file +5. **Recalculate formulas (MANDATORY IF USING FORMULAS)**: Use the scripts/recalc.py script + ```bash + python3 scripts/recalc.py output.xlsx + ``` +6. **Verify and fix any errors**: + - The script returns JSON with error details + - If `status` is `errors_found`, check `error_summary` for specific error types and locations + - Fix the identified errors and recalculate again + - Common errors to fix: + - `#REF!`: Invalid cell references + - `#DIV/0!`: Division by zero + - `#VALUE!`: Wrong data type in formula + - `#NAME?`: Unrecognized formula name + +### Creating new Excel files + +```python +# Using openpyxl for formulas and formatting +from openpyxl import Workbook +from openpyxl.styles import Font, PatternFill, Alignment + +wb = Workbook() +sheet = wb.active + +# Add data +sheet['A1'] = 'Hello' +sheet['B1'] = 'World' +sheet.append(['Row', 'of', 'data']) + +# Add formula +sheet['B2'] = '=SUM(A1:A10)' + +# Formatting +sheet['A1'].font = Font(bold=True, color='FF0000') +sheet['A1'].fill = PatternFill('solid', start_color='FFFF00') +sheet['A1'].alignment = Alignment(horizontal='center') + +# Column width +sheet.column_dimensions['A'].width = 20 + +wb.save('output.xlsx') +``` + +### Editing existing Excel files + +```python +# Using openpyxl to preserve formulas and formatting +from openpyxl import load_workbook + +# Load existing file +wb = load_workbook('existing.xlsx') +sheet = wb.active # or wb['SheetName'] for specific sheet + +# Working with multiple sheets +for sheet_name in wb.sheetnames: + sheet = wb[sheet_name] + print(f"Sheet: {sheet_name}") + +# Modify cells +sheet['A1'] = 'New Value' +sheet.insert_rows(2) # Insert row at position 2 +sheet.delete_cols(3) # Delete column 3 + +# Add new sheet +new_sheet = wb.create_sheet('NewSheet') +new_sheet['A1'] = 'Data' + +wb.save('modified.xlsx') +``` + +## Recalculating formulas + +Excel files created or modified by openpyxl contain formulas as strings but not calculated values. Use the provided `scripts/recalc.py` script to recalculate formulas: + +```bash +python3 scripts/recalc.py [timeout_seconds] +``` + +Example: +```bash +python3 scripts/recalc.py output.xlsx 30 +``` + +The script: +- Automatically sets up LibreOffice macro on first run +- Recalculates all formulas in all sheets +- Scans ALL cells for Excel errors (#REF!, #DIV/0!, etc.) +- Returns JSON with detailed error locations and counts +- Works on both Linux and macOS + +## Formula Verification Checklist + +Quick checks to ensure formulas work correctly: + +### Essential Verification +- [ ] **Test 2-3 sample references**: Verify they pull correct values before building full model +- [ ] **Column mapping**: Confirm Excel columns match (e.g., column 64 = BL, not BK) +- [ ] **Row offset**: Remember Excel rows are 1-indexed (DataFrame row 5 = Excel row 6) + +### Common Pitfalls +- [ ] **NaN handling**: Check for null values with `pd.notna()` +- [ ] **Far-right columns**: FY data often in columns 50+ +- [ ] **Multiple matches**: Search all occurrences, not just first +- [ ] **Division by zero**: Check denominators before using `/` in formulas (#DIV/0!) +- [ ] **Wrong references**: Verify all cell references point to intended cells (#REF!) +- [ ] **Cross-sheet references**: Use correct format (Sheet1!A1) for linking sheets + +### Formula Testing Strategy +- [ ] **Start small**: Test formulas on 2-3 cells before applying broadly +- [ ] **Verify dependencies**: Check all cells referenced in formulas exist +- [ ] **Test edge cases**: Include zero, negative, and very large values + +### Interpreting scripts/recalc.py Output +The script returns JSON with error details: +```json +{ + "status": "success", // or "errors_found" + "total_errors": 0, // Total error count + "total_formulas": 42, // Number of formulas in file + "error_summary": { // Only present if errors found + "#REF!": { + "count": 2, + "locations": ["Sheet1!B5", "Sheet1!C10"] + } + } +} +``` + +## Best Practices + +### Library Selection +- **pandas**: Best for data analysis, bulk operations, and simple data export +- **openpyxl**: Best for complex formatting, formulas, and Excel-specific features + +### Working with openpyxl +- Cell indices are 1-based (row=1, column=1 refers to cell A1) +- Use `data_only=True` to read calculated values: `load_workbook('file.xlsx', data_only=True)` +- **Warning**: If opened with `data_only=True` and saved, formulas are replaced with values and permanently lost +- For large files: Use `read_only=True` for reading or `write_only=True` for writing +- Formulas are preserved but not evaluated - use scripts/recalc.py to update values + +### Working with pandas +- Specify data types to avoid inference issues: `pd.read_excel('file.xlsx', dtype={'id': str})` +- For large files, read specific columns: `pd.read_excel('file.xlsx', usecols=['A', 'C', 'E'])` +- Handle dates properly: `pd.read_excel('file.xlsx', parse_dates=['date_column'])` + +## Code Style Guidelines +**IMPORTANT**: When generating Python code for Excel operations: +- Write minimal, concise Python code without unnecessary comments +- Avoid verbose variable names and redundant operations +- Avoid unnecessary print statements + +**For Excel files themselves**: +- Add comments to cells with complex formulas or important assumptions +- Document data sources for hardcoded values +- Include notes for key calculations and model sections