Complete End-to-End Pipeline Integration with Error Recovery and Testing

Summary

Complete the integration of all StringyMcStringFace components into a cohesive end-to-end pipeline with comprehensive error recovery mechanisms and full integration test coverage.

Context

StringyMcStringFace is currently in a state where individual components are implemented or framework-ready:

✅ Format detection (ELF, PE, Mach-O via goblin)
✅ Container parsers with section classification
✅ Import/export extraction
🚧 String extraction engines (ASCII/UTF-8, UTF-16)
🚧 Semantic classification (URLs, paths, GUIDs, etc.)
🚧 Ranking and scoring system
🚧 Output formatters (JSON, human-readable, YARA)

The pipeline integration task involves connecting these components into a complete data flow from binary input to formatted output.

Pipeline Architecture

Binary Input
    ↓
Format Detection (goblin)
    ↓
Container Parser Selection (ELF/PE/Mach-O)
    ↓
Section Classification & Weighting
    ↓
String Extraction (ASCII/UTF-8/UTF-16)
    ↓
Semantic Classification & Tagging
    ↓
Scoring & Ranking
    ↓
Output Formatting (JSON/Text/YARA)
    ↓
Final Output

Proposed Solution

1. Pipeline Orchestration Module

Create a pipeline.rs module that coordinates the flow:

Pipeline struct: Central orchestrator that manages state and error context
PipelineStage enum: Track which stage failed for better error messages
PipelineResult: Accumulates results and partial failures throughout execution
run_pipeline(): Main entry point that executes all stages in sequence

2. Error Recovery Strategy

Implement graceful degradation at each stage:

Format Detection Failure: Fall back to raw binary analysis mode
Parser Errors: Skip corrupted sections, continue with valid ones
Extraction Failures: Log warnings, continue with other encodings
Classification Errors: Mark strings as unclassified but retain them
Scoring Failures: Use default score (50) as fallback

Error handling pattern:

match stage_result {
    Ok(data) => pipeline.add_results(data),
    Err(e) => {
        pipeline.log_error(stage, e);
        pipeline.continue_with_partial();
    }
}

3. Data Flow Integration

Ensure consistent data structures flow between stages:

Stage 1→2: BinaryFormat → ContainerParser
Stage 2→3: Section metadata → StringExtractor
Stage 3→4: RawString → Classifier
Stage 4→5: TaggedString → Scorer
Stage 5→6: ScoredString → OutputFormatter

4. End-to-End Integration Tests

Create comprehensive test suite in tests/integration/:

Test Fixtures

test_binary_elf: ELF with known strings in .rodata
test_binary_pe.exe: PE with UTF-16 strings and resources
test_binary_macho: Mach-O with __cstring section
corrupted_binary: Intentionally malformed for error recovery tests

Test Categories

Happy Path Tests:

test_elf_complete_pipeline(): Verify full ELF analysis
test_pe_complete_pipeline(): Verify full PE analysis with UTF-16
test_macho_complete_pipeline(): Verify full Mach-O analysis
test_json_output_format(): Validate JSON structure
test_semantic_tagging(): Verify URLs, paths, GUIDs detected

Error Recovery Tests:

test_corrupted_section_recovery(): Skip bad sections, continue analysis
test_unknown_format_fallback(): Handle unrecognized formats gracefully
test_partial_utf16_recovery(): Handle incomplete UTF-16 sequences
test_empty_sections_handling(): Process binaries with no string data

Performance Tests:

test_large_binary_performance(): Ensure reasonable performance on large binaries
test_memory_limits(): Verify memory usage stays within bounds

5. Configuration & CLI Integration

Update CLI to support pipeline configuration:

--max-errors N: Stop after N errors (default: continue all)
--skip-sections PATTERN: Exclude sections matching pattern
--strict: Fail fast on any error (no recovery)
--verbose: Show pipeline stage progress

Acceptance Criteria

Implementation Checklist

Dependencies

Blocked by: Main Extraction Pipeline implementation
Depends on: String extraction engines must be functional
Depends on: Semantic classification system must be operational

Related Issues

Reference any related issues for string extraction, classification, or scoring components.

Task ID

stringy-analyzer/complete-pipeline-integration

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

@unclesp1d3r

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch current issue #37 details
gh issue view 37 --json title,body,labels,assignees,milestone,state,url

Length of output: 1454

🏁 Script executed:

#!/bin/bash
# Search for pipeline-related files in the codebase
fd -e py -e js -e ts | head -30

Length of output: 58

🏁 Script executed:

#!/bin/bash
# Search for pipeline integration code
rg -n -C3 "pipeline" --type py --type js --type ts | head -100

Length of output: 6944

🏁 Script executed:

#!/bin/bash
# Look for test files and integration tests
fd test | head -20

Length of output: 133

🏁 Script executed:

#!/bin/bash
# Check for README or documentation
cat README.md 2>/dev/null || echo "No README.md found"

Length of output: 7196

🏁 Script executed:

#!/bin/bash
# Update issue #37 with enhanced content

gh issue edit 37 \
  --title "Complete End-to-End Pipeline Integration with Error Recovery and Testing" \
  --body "## Summary

Complete the integration of all StringyMcStringFace components into a cohesive end-to-end pipeline with comprehensive error recovery mechanisms and full integration test coverage.

## Context

StringyMcStringFace is currently in a state where individual components are implemented or framework-ready:
- ✅ Format detection (ELF, PE, Mach-O via goblin)
- ✅ Container parsers with section classification
- ✅ Import/export extraction
- 🚧 String extraction engines (ASCII/UTF-8, UTF-16)
- 🚧 Semantic classification (URLs, paths, GUIDs, etc.)
- 🚧 Ranking and scoring system
- 🚧 Output formatters (JSON, human-readable, YARA)

The pipeline integration task involves connecting these components into a complete data flow from binary input to formatted output.

## Pipeline Architecture

\`\`\`
Binary Input
    ↓
Format Detection (goblin)
    ↓
Container Parser Selection (ELF/PE/Mach-O)
    ↓
Section Classification & Weighting
    ↓
String Extraction (ASCII/UTF-8/UTF-16)
    ↓
Semantic Classification & Tagging
    ↓
Scoring & Ranking
    ↓
Output Formatting (JSON/Text/YARA)
    ↓
Final Output
\`\`\`

## Proposed Solution

### 1. Pipeline Orchestration Module

Create a \`pipeline.rs\` module that coordinates the flow:

- **\`Pipeline\` struct**: Central orchestrator that manages state and error context
- **\`PipelineStage\` enum**: Track which stage failed for better error messages
- **\`PipelineResult\`**: Accumulates results and partial failures throughout execution
- **\`run_pipeline()\`**: Main entry point that executes all stages in sequence

### 2. Error Recovery Strategy

Implement graceful degradation at each stage:

- **Format Detection Failure**: Fall back to raw binary analysis mode
- **Parser Errors**: Skip corrupted sections, continue with valid ones
- **Extraction Failures**: Log warnings, continue with other encodings
- **Classification Errors**: Mark strings as unclassified but retain them
- **Scoring Failures**: Use default score (50) as fallback

Error handling pattern:
\`\`\`rust
match stage_result {
    Ok(data) => pipeline.add_results(data),
    Err(e) => {
        pipeline.log_error(stage, e);
        pipeline.continue_with_partial();
    }
}
\`\`\`

### 3. Data Flow Integration

Ensure consistent data structures flow between stages:

- **Stage 1→2**: \`BinaryFormat\` → \`ContainerParser\`
- **Stage 2→3**: \`Section\` metadata → \`StringExtractor\`
- **Stage 3→4**: \`RawString\` → \`Classifier\`
- **Stage 4→5**: \`TaggedString\` → \`Scorer\`
- **Stage 5→6**: \`ScoredString\` → \`OutputFormatter\`

### 4. End-to-End Integration Tests

Create comprehensive test suite in \`tests/integration/\`:

#### Test Fixtures
- \`test_binary_elf\`: ELF with known strings in .rodata
- \`test_binary_pe.exe\`: PE with UTF-16 strings and resources
- \`test_binary_macho\`: Mach-O with __cstring section
- \`corrupted_binary\`: Intentionally malformed for error recovery tests

#### Test Categories

**Happy Path Tests**:
- \`test_elf_complete_pipeline()\`: Verify full ELF analysis
- \`test_pe_complete_pipeline()\`: Verify full PE analysis with UTF-16
- \`test_macho_complete_pipeline()\`: Verify full Mach-O analysis
- \`test_json_output_format()\`: Validate JSON structure
- \`test_semantic_tagging()\`: Verify URLs, paths, GUIDs detected

**Error Recovery Tests**:
- \`test_corrupted_section_recovery()\`: Skip bad sections, continue analysis
- \`test_unknown_format_fallback()\`: Handle unrecognized formats gracefully
- \`test_partial_utf16_recovery()\`: Handle incomplete UTF-16 sequences
- \`test_empty_sections_handling()\`: Process binaries with no string data

**Performance Tests**:
- \`test_large_binary_performance()\`: Ensure reasonable performance on large binaries
- \`test_memory_limits()\`: Verify memory usage stays within bounds

### 5. Configuration & CLI Integration

Update CLI to support pipeline configuration:
- \`--max-errors N\`: Stop after N errors (default: continue all)
- \`--skip-sections PATTERN\`: Exclude sections matching pattern
- \`--strict\`: Fail fast on any error (no recovery)
- \`--verbose\`: Show pipeline stage progress

## Acceptance Criteria

- [ ] Pipeline orchestration module implemented with all stages connected
- [ ] Error recovery mechanisms in place for all failure modes
- [ ] At least 15 integration tests covering happy path and error scenarios
- [ ] All test fixtures (\`test_binary_elf\`, \`test_binary_pe.exe\`, \`test_binary_macho\`) processed successfully
- [ ] JSON output validates against expected schema
- [ ] Semantic tagging correctly identifies URLs, paths, GUIDs in test binaries
- [ ] CLI successfully processes real-world binaries (\`/bin/ls\`, example PE file)
- [ ] Error messages include pipeline stage context and recovery actions taken
- [ ] Performance: Process 10MB binary in < 5 seconds on typical hardware
- [ ] Documentation updated with pipeline architecture and error handling strategy
- [ ] Code coverage > 80% for pipeline module

## Implementation Checklist

- [ ] Create \`src/pipeline.rs\` module
- [ ] Define \`Pipeline\`, \`PipelineStage\`, \`PipelineResult\` types
- [ ] Implement \`run_pipeline()\` orchestration function
- [ ] Add error recovery logic for each stage
- [ ] Create \`tests/integration/pipeline_tests.rs\`
- [ ] Generate test fixtures with known string content
- [ ] Write happy path integration tests (3 formats × 2 output modes)
- [ ] Write error recovery tests (5+ scenarios)
- [ ] Add CLI flags for pipeline configuration
- [ ] Update main.rs to use pipeline orchestrator
- [ ] Add pipeline architecture diagram to docs
- [ ] Write troubleshooting guide for common errors

## Dependencies

- **Blocked by**: Main Extraction Pipeline implementation
- **Depends on**: String extraction engines must be functional
- **Depends on**: Semantic classification system must be operational

## Related Issues

Reference any related issues for string extraction, classification, or scoring components.

## Task ID

\`stringy-analyzer/complete-pipeline-integration\`"

Length of output: 61

🏁 Script executed:

#!/bin/bash
# Add additional labels for priority and complexity
gh issue edit 37 --add-label "priority:high" 2>/dev/null || echo "Label priority:high may not exist, skipping"
gh issue edit 37 --add-label "complexity:high" 2>/dev/null || echo "Label complexity:high may not exist, skipping"

Length of output: 107

🏁 Script executed:

#!/bin/bash
# Show the updated issue
gh issue view 37 --json title,body,labels,assignees,milestone,state | jq

Length of output: 7468

@unclesp1d3r I've successfully enhanced issue #37! Here's what I updated:

✅ Title Enhanced

Before: "Complete Pipeline Integration"
After: "Complete End-to-End Pipeline Integration with Error Recovery and Testing"

✅ Body Enhanced with:

1. Detailed Context

Current implementation status of all components
Clear explanation of what pipeline integration entails

2. Visual Pipeline Architecture

Shows the complete data flow from binary input through all 7 stages to final output

3. Comprehensive Proposed Solution

Pipeline Orchestration Module: Defines Pipeline struct, PipelineStage enum, and run_pipeline() function
Error Recovery Strategy: Graceful degradation at each stage with fallback behaviors
Data Flow Integration: Type-safe data structures flowing between stages
End-to-End Integration Tests: 15+ specific test cases covering:
- Happy path tests (ELF, PE, Mach-O formats)
- Error recovery tests (corrupted sections, unknown formats, partial UTF-16)
- Performance tests (large binaries, memory limits)
CLI Integration: New flags like --max-errors, --skip-sections, --strict, --verbose

4. Measurable Acceptance Criteria

11 specific checkboxes including performance targets (10MB binary < 5s) and code coverage goals (>80%)

5. Implementation Checklist

12 actionable items from creating src/pipeline.rs to writing troubleshooting guides

✅ Labels Updated

Added: priority:high ✓
Retained: area:analyzer, type:enhancement, status:backlog, lang:rust, needs:tests
Note: complexity:high label doesn't exist in the repository

🔗 View Updated Issue

#37

The issue now provides a complete technical roadmap for implementing the pipeline integration with specific code structures, test scenarios, and success metrics!

Plan

Observations

Current State Analysis

The StringyMcStringFace project has a solid foundation with fully implemented container parsing for ELF, PE, and Mach-O formats. The container parsers provide comprehensive section classification, import/export extraction, and intelligent weighting systems. However, the core pipeline components (string extraction, classification, ranking, output) are empty stubs.

What's Complete:

✅ Container parsing with sophisticated section classification and weighting
✅ Import/export symbol extraction for all three formats
✅ Core data types (FoundString, ContainerInfo, Tag, Encoding, etc.)
✅ Error handling framework with StringyError enum
✅ Integration test infrastructure with binary fixtures
✅ CLI argument parsing structure (basic)

What's Missing:

❌ String extraction engines (ASCII/UTF-8, UTF-16)
❌ Semantic classification system (pattern matching, tagging)
❌ Ranking/scoring algorithm implementation
❌ Output formatters (JSON, human-readable, YARA)
❌ Pipeline orchestration module
❌ CLI integration with pipeline

Critical Dependencies:
The GitHub issue explicitly states this task depends on string extraction engines and semantic classification being functional. Since these don't exist yet, this plan must implement them first before the pipeline integration can occur.

Architecture Insights:

The design documents specify a clear pipeline: Format Detection → Container Parsing → String Extraction → Classification → Ranking → Output
Section weights (1.0-10.0) are already calculated by parsers
The scoring formula is defined: Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty
Test fixtures exist with known strings ("Hello, world!\n", "Helper called\n")

Approach

This implementation follows a bottom-up approach, building the missing pipeline components before integrating them:

String Extraction Layer - Implement ASCII/UTF-8 and UTF-16 extraction engines with configurable parameters
Classification Layer - Build semantic pattern matching and symbol processing
Ranking Layer - Implement the scoring algorithm with section weights, semantic boosts, and noise penalties
Output Layer - Create formatters for JSON, human-readable, and YARA outputs
Pipeline Orchestration - Wire all components together with error recovery
CLI Integration - Connect the pipeline to command-line arguments
Integration Testing - Comprehensive end-to-end tests with error recovery scenarios

The approach prioritizes graceful degradation at each stage, allowing partial results even when individual components fail. Each module will be self-contained with clear interfaces, making the system testable and maintainable.

Reasoning

I explored the repository structure and identified that container parsing is fully implemented while extraction, classification, ranking, and output modules are empty. I examined the ELF, PE, and Mach-O parsers to understand the section weighting system, reviewed the design documents to understand the intended architecture, checked the test fixtures to see what strings should be extracted, and analyzed the CLI documentation to understand the expected interface. The issue description provided the pipeline architecture and error recovery requirements.

Proposed File Changes

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/types.rs

Create the string extraction framework with traits and core types:

Define StringExtractor trait with extract_from_section() method that takes section data and metadata, returns vector of extracted strings

Define RawString struct to hold extracted string data before classification: text (String), offset (u64), encoding (Encoding), section_name (Option), source (StringSource)

Define ExtractionConfig struct with configurable parameters: min_ascii_length (default 4), min_wide_length (default 3), enabled_encodings (Vec), max_string_length (default 1024)

Implement helper function extract_strings_from_section() that dispatches to appropriate extractor based on encoding configuration

Add module declarations for ascii and utf16 submodules

This provides the foundation for string extraction that will be used by the pipeline orchestrator.

📄 src/extraction/ascii.rs ^(NEW) 🔗

References

📄 src/extraction/mod.rs ^(MODIFY)

📄 src/types.rs

Implement ASCII/UTF-8 string extraction:

Create AsciiExtractor struct with configurable minimum length

Implement StringExtractor trait for AsciiExtractor

Implement extract_from_section() to scan byte sequences for printable ASCII characters (0x20-0x7E, plus common extended characters like tabs and newlines)

Extract null-terminated strings and sequences of printable characters

Filter out obvious noise patterns: repeated padding bytes (0x00, 0xFF), table-like data with regular patterns

Consider section context from SectionInfo to apply appropriate heuristics (e.g., more lenient in .rodata vs .data)

Return RawString instances with proper offset, encoding (Ascii or Utf8), and section metadata

Add unit tests for basic extraction, noise filtering, and edge cases (empty sections, all-noise data, mixed content)

This implements the primary string extraction mechanism for ASCII-encoded strings.

📄 src/extraction/utf16.rs ^(NEW) 🔗

References

📄 src/extraction/mod.rs ^(MODIFY)

📄 src/types.rs

Implement UTF-16 string extraction:

Create Utf16Extractor struct with configurable minimum length and confidence threshold

Implement StringExtractor trait for Utf16Extractor

Implement extract_from_section() to detect UTF-16LE and UTF-16BE encoded strings

For UTF-16LE: scan for even-length sequences where low bytes are mostly printable (0x20-0x7E) and high bytes are mostly zero

For UTF-16BE: similar logic but with byte order reversed

Calculate confidence score based on ratio of valid characters to total length

Handle null-interleaved text patterns (ASCII with null bytes between characters)

Filter low-confidence extractions to avoid false positives

Return RawString instances with appropriate encoding (Utf16Le or Utf16Be) and confidence metadata

Add unit tests for UTF-16LE extraction, UTF-16BE extraction, null-interleaved patterns, and confidence scoring

This enables extraction of wide-character strings common in Windows PE binaries.

📄 src/extraction/dedup.rs ^(NEW) 🔗

References

📄 src/extraction/mod.rs ^(MODIFY)

📄 src/types.rs

Implement string deduplication:

Create deduplicate_strings() function that takes vector of RawString and returns deduplicated vector

Use HashMap with string text as key to identify duplicates

For duplicates, preserve the first occurrence's metadata (offset, section) but track all occurrences

Implement canonicalization: trim whitespace, normalize line endings, but preserve original for output

Maintain a list of all offsets where each unique string appears

Add unit tests for basic deduplication, metadata preservation, and handling of strings that appear in multiple sections

This reduces noise in output while preserving provenance information.

📄 src/classification/mod.rs ^(MODIFY) 🔗

References

📄 src/types.rs

Create the classification framework:

Define Classifier trait with classify() method that takes string text and context, returns vector of Tag

Define StringContext struct with section information, source type, and encoding to provide context for classification

Create SemanticClassifier struct that holds compiled regex patterns for URL, domain, IP, path, GUID, email, Base64, format string, and user agent detection

Implement lazy initialization of regex patterns using once_cell or similar to cache compiled patterns

Implement classify_string() function that applies all pattern matchers and returns matching tags

Add helper function classify_import_export() that tags strings from import/export sources

Add module declarations for semantic and symbols submodules

Add unit tests for basic classification and context handling

This provides the foundation for semantic tagging of extracted strings.

📄 src/classification/semantic.rs ^(NEW) 🔗

References

📄 src/classification/mod.rs ^(MODIFY)

📄 src/types.rs

Implement semantic pattern matching:

Create regex patterns for each semantic category:

URL: https?://[^\s]+ with validation

Domain: [a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+

IPv4: \b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b

IPv6: standard IPv6 pattern with hex groups

File path (POSIX): /[a-zA-Z0-9_/.\-]+

File path (Windows): [A-Z]:\\[^\s]+ or \\\\[^\s]+

Registry path: HKEY_[A-Z_]+\\[^\s]+

GUID: \{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}

Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

Base64: [A-Za-z0-9+/]{20,}={0,2} (minimum length to reduce false positives)

Format string: %[0-9]*[sdxfp] patterns

User agent: strings containing "Mozilla", "Chrome", "Safari", etc.

Implement PatternMatcher struct that caches compiled regexes

Implement match_patterns() function that applies all patterns and returns matching tags

Add validation logic to reduce false positives (e.g., validate IPv4 octets are 0-255)

Add unit tests for each pattern type with positive and negative cases

This implements the core semantic classification logic.

📄 src/classification/symbols.rs ^(NEW) 🔗

References

📄 src/classification/mod.rs ^(MODIFY)

📄 src/types.rs

Implement symbol processing:

Add rustc-demangle dependency to Cargo.toml if not present

Create SymbolProcessor struct for handling symbol names

Implement demangle_rust_symbol() function using rustc_demangle::demangle()

Implement process_import_name() that tags import symbols with Tag::Import and applies demangling if applicable

Implement process_export_name() that tags export symbols with Tag::Export and applies demangling

Handle common symbol prefixes (underscore for C symbols, mangled C++ names)

Add unit tests for Rust symbol demangling, import/export tagging, and handling of various symbol formats

This enables proper handling of symbol names from import/export tables.

📄 src/classification/ranking.rs ^(NEW) 🔗

References

📄 src/types.rs

Implement the ranking and scoring algorithm:

Create RankingEngine struct with configurable weights and penalties

Define RankingConfig with section weight multipliers, tag boost values, and noise penalty thresholds

Implement calculate_score() function that computes: Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty

Section weight component: use the weight from SectionInfo (already calculated by parsers, 1.0-10.0) and multiply by 10 to get base score

Encoding confidence: +10 for clean ASCII/UTF-8, +15 for high-confidence UTF-16, +5 for low-confidence

Semantic boost: URL/GUID/registry (+50), paths/format strings (+30), imports/exports (+20), other tags (+10)

Noise penalty: high entropy (-30), excessive length >200 chars (-20), repeated patterns (-10), obvious table data (-15)

Implement calculate_entropy() helper to detect high-entropy strings

Implement detect_repeated_patterns() to identify padding or table data

Implement is_table_data() heuristic based on regular spacing and similar characters

Ensure final score is clamped to reasonable range (0-100)

Add unit tests for score calculation with various combinations of factors

This implements the core ranking algorithm to prioritize relevant strings.

📄 src/output/mod.rs ^(MODIFY) 🔗

References

📄 src/types.rs

Create the output formatting framework:

Define OutputFormatter trait with format() method that takes vector of FoundString and returns formatted string

Define OutputFormat enum with variants: Human, Json, Yara

Define OutputConfig struct with format selection, color settings, truncation limits

Implement format_output() function that dispatches to appropriate formatter based on config

Add module declarations for json, human, and yara submodules

Add helper functions for common formatting tasks (escaping, truncation, color codes)

This provides the foundation for multiple output formats.

📄 src/output/json.rs ^(NEW) 🔗

References

📄 src/output/mod.rs ^(MODIFY)

📄 src/types.rs

Implement JSON Lines output formatter:

Create JsonFormatter struct implementing OutputFormatter trait

Implement format() to serialize each FoundString as a JSON object on a single line

Use serde_json::to_string() for serialization (FoundString already has Serialize derive)

Ensure output is valid JSONL format (one JSON object per line, no trailing comma)

Include all fields: text, encoding, offset, rva, section, length, tags, score, source

Handle null values appropriately (rva and section can be None)

Add unit tests for formatting single strings, multiple strings, and strings with various field combinations

This implements the machine-readable output format for pipeline integration.

📄 src/output/human.rs ^(NEW) 🔗

References

📄 src/output/mod.rs ^(MODIFY)

📄 src/types.rs

Implement human-readable table output formatter:

Create HumanFormatter struct implementing OutputFormatter trait with color support configuration

Implement format() to create aligned table with columns: Score, Offset, Section, Encoding, Tags, String

Calculate column widths dynamically based on content

Truncate long strings with "..." indicator (default 60 characters)

Apply color coding based on score: green (>80), yellow (50-80), red (<50) using ANSI escape codes

Respect NO_COLOR environment variable to disable colors

Format offset as hexadecimal (0x prefix)

Join multiple tags with commas

Add table header with separator line

Add unit tests for formatting, color coding, truncation, and column alignment

This implements the interactive output format for manual analysis.

📄 src/output/yara.rs ^(NEW) 🔗

References

📄 src/output/mod.rs ^(MODIFY)

📄 src/types.rs

Implement YARA rule output formatter:

Create YaraFormatter struct implementing OutputFormatter trait

Implement format() to generate valid YARA rule syntax

Create rule header with metadata: description, generated_by, timestamp

Group strings by semantic category (URLs, GUIDs, paths, etc.)

Escape special characters in strings: backslash, quotes, newlines, tabs

Add appropriate modifiers: ascii for ASCII strings, wide for UTF-16, ascii wide for both

Convert binary data to hex format when necessary

Add comments with score and classification information

Generate simple condition: any of them

Truncate very long strings (>100 chars) with comment indicating truncation

Add unit tests for escaping, grouping, modifier selection, and complete rule generation

This implements the YARA-friendly output format for security rule creation.

📄 src/pipeline.rs ^(NEW) 🔗

References

📄 src/types.rs

📄 src/container/mod.rs

📄 src/extraction/mod.rs ^(MODIFY)

📄 src/classification/mod.rs ^(MODIFY)

📄 src/output/mod.rs ^(MODIFY)

Create the pipeline orchestration module:

Define Pipeline struct that holds configuration and accumulated state

Define PipelineStage enum with variants: FormatDetection, ContainerParsing, StringExtraction, Classification, Ranking, OutputFormatting

Define PipelineResult struct that accumulates results and tracks errors: strings (Vec), errors (Vec), warnings (Vec), metadata (PipelineMetadata)

Define PipelineError struct with stage (PipelineStage), error (StringyError), recovery_action (String)

Define PipelineMetadata struct with format (BinaryFormat), sections_analyzed (Vec), total_strings_found (usize), extraction_time_ms (u64)

Define PipelineConfig struct with extraction_config (ExtractionConfig), ranking_config (RankingConfig), output_config (OutputConfig), max_errors (Option), strict_mode (bool)

Implement Pipeline::new() constructor that takes config

Implement Pipeline::run() main orchestration function that:

Detects format using container::detect_format()

Creates parser using container::create_parser()

Parses container to get ContainerInfo

Extracts strings from sections using extraction engines

Adds import/export names as strings

Deduplicates strings

Classifies strings with semantic tags

Calculates scores using ranking engine

Sorts by score descending

Formats output

Implement error recovery at each stage:

Format detection failure: return error (cannot proceed without format)

Parser errors: log error, try to continue with partial section data

Extraction failures: log warning, skip problematic section, continue with others

Classification errors: log warning, mark string as unclassified, continue

Scoring failures: log warning, use default score (50), continue

Output formatting errors: return error (final stage, must succeed)

Implement Pipeline::log_error() to track errors with stage context

Implement Pipeline::log_warning() to track non-fatal issues

Implement Pipeline::should_continue() to check if max_errors threshold reached

Add comprehensive error messages with recovery actions taken

This implements the core pipeline orchestration with graceful degradation.

📄 src/lib.rs ^(MODIFY) 🔗

References

📄 src/pipeline.rs ^(NEW)

Update library exports to include new modules:

Add pub mod pipeline; declaration

Re-export pipeline types: Pipeline, PipelineConfig, PipelineResult, PipelineStage

Update documentation to reflect that pipeline integration is now complete

Update implementation status comments to mark extraction, classification, ranking, and output as complete

This makes the pipeline module accessible to external users of the library.

📄 src/main.rs ^(MODIFY) 🔗

References

📄 src/pipeline.rs ^(NEW)

📄 src/types.rs

Implement the CLI integration with pipeline:

Expand Cli struct to include all command-line arguments:

input: PathBuf (already exists)

min_len: Option<usize> with default 4

max_len: Option<usize> with default 1024

enc: Option<String> for comma-separated encodings

only_tags: Option<String> for tag filtering

exclude_tags: Option<String> for tag exclusion

sections: Option<String> for section filtering

top: Option<usize> for result limiting (default 100)

all: bool flag to show all results

format: Option<String> with values: human, json, yara

json: bool shorthand flag

yara: bool shorthand flag

max_errors: Option<usize> for error threshold

strict: bool for fail-fast mode

verbose: bool for pipeline stage progress

Implement main() function:

Parse CLI arguments using clap

Read binary file using std::fs::read()

Build PipelineConfig from CLI arguments

Create Pipeline instance with config

Run pipeline with pipeline.run(&data)

Handle PipelineResult: print warnings/errors if verbose, output formatted strings

Return appropriate exit code based on errors

Implement helper function parse_encodings() to convert comma-separated string to Vec

Implement helper function parse_tags() to convert comma-separated string to Vec

Add error handling for file I/O errors with clear messages

Add progress indicators if verbose flag is set

This connects the CLI to the pipeline orchestrator.

📄 Cargo.toml ^(MODIFY) 🔗

Add required dependencies for new functionality:

Add regex = "1.10" for pattern matching in semantic classification

Add rustc-demangle = "0.1" for Rust symbol demangling

Add once_cell = "1.19" for lazy static regex compilation

Consider adding memmap2 = "0.9" for memory-mapped file I/O (optional, can be added later for performance)

These dependencies are needed for the classification and symbol processing modules.

📄 tests/integration/pipeline_tests.rs ^(NEW) 🔗

References

📄 src/pipeline.rs ^(NEW)

📄 tests/fixtures/test_binary_elf

📄 tests/fixtures/test_binary_pe.exe

📄 tests/fixtures/test_binary_macho

Create comprehensive integration tests for the pipeline:

Happy Path Tests:

test_elf_complete_pipeline(): Load test_binary_elf fixture, run full pipeline, verify strings extracted ("Hello, world!", "Helper called"), verify imports found (printf, malloc, free), verify exports found (main, exported_function, helper_function), verify scores assigned, verify output formats work

test_pe_complete_pipeline(): Load test_binary_pe.exe fixture, run full pipeline, verify PE-specific handling, verify UTF-16 extraction if applicable, verify section classification

test_macho_complete_pipeline(): Load test_binary_macho fixture, run full pipeline, verify Mach-O section handling, verify symbol extraction

test_json_output_format(): Run pipeline with JSON output, parse output, validate JSON structure matches schema, verify all required fields present

test_human_output_format(): Run pipeline with human-readable output, verify table formatting, verify column alignment

test_yara_output_format(): Run pipeline with YARA output, verify valid YARA syntax, verify proper escaping

test_semantic_tagging(): Create test binary with known URLs, paths, GUIDs, verify they are correctly tagged

Error Recovery Tests:

test_corrupted_section_recovery(): Create binary with intentionally corrupted section data, verify pipeline continues with other sections, verify error logged but partial results returned

test_unknown_format_fallback(): Provide non-binary data, verify graceful error handling with clear message

test_partial_utf16_recovery(): Create data with incomplete UTF-16 sequences, verify extractor handles gracefully, continues with valid strings

test_empty_sections_handling(): Create binary with zero-size sections, verify pipeline skips them without errors

test_max_errors_threshold(): Configure pipeline with max_errors=3, trigger multiple errors, verify pipeline stops after threshold

test_strict_mode_failure(): Enable strict mode, trigger error, verify pipeline fails immediately

Performance Tests:

test_large_binary_performance(): Create or use large binary (10MB+), verify processing completes in reasonable time (<5 seconds), verify memory usage reasonable

test_many_strings_handling(): Create binary with thousands of strings, verify deduplication works, verify ranking performance

Configuration Tests:

test_encoding_filtering(): Configure pipeline to extract only ASCII, verify UTF-16 strings not extracted

test_section_filtering(): Configure pipeline to scan only .rodata, verify other sections skipped

test_tag_filtering(): Configure pipeline to show only URL tags, verify other tags filtered from output

test_top_n_limiting(): Configure pipeline with top=10, verify only 10 highest-scoring strings returned

Use insta for snapshot testing where appropriate. Each test should verify both success conditions and proper error handling.

📁 tests/integration ^(NEW) 🔗

Create integration test directory to organize end-to-end tests separately from unit tests. This directory will contain pipeline_tests.rs and potentially other integration test files in the future.

📄 docs/src/architecture.md ^(MODIFY) 🔗

References

📄 src/pipeline.rs ^(NEW)

Update architecture documentation to reflect completed implementation:

Change status indicators from 🚧 to ✅ for Extraction, Classification, Ranking, and Output modules

Add section describing the Pipeline module with diagram showing data flow through stages

Document error recovery strategy with examples of each recovery scenario

Add section on PipelineConfig and how it controls behavior

Update "Current Implementation Details" section to include extraction, classification, ranking, and output implementations

Add examples of pipeline usage from code

Update module dependency diagram to include pipeline.rs

This keeps documentation in sync with implementation.

📄 docs/src/troubleshooting.md ^(MODIFY) 🔗

References

📄 src/pipeline.rs ^(NEW)

Add troubleshooting guide for pipeline errors:

Add section "Pipeline Error Recovery" explaining how the pipeline handles errors at each stage

Document common error scenarios:

"Unsupported file format" - what to check, how to verify file type

"Section parsing failed" - how pipeline recovers, what partial results to expect

"String extraction errors" - encoding issues, how to adjust min-len

"Classification failures" - regex compilation errors, how to disable problematic patterns

"Output formatting errors" - invalid characters, escaping issues

Add section "Debugging Pipeline Issues" with --verbose flag usage

Add section "Performance Issues" with tips for large binaries

Add examples of error messages and their meanings

Add section "Strict Mode vs. Graceful Degradation" explaining when to use each

This helps users understand and resolve pipeline-related issues.

📄 README.md ^(MODIFY) 🔗

Update README to reflect completed implementation:

Change "Current Status" section to mark all components as ✅ Complete

Update "Usage" section with actual working examples using the implemented CLI flags

Add "Quick Start" section with common use cases:

Basic analysis: stringy binary

JSON output: stringy --json binary

Filter by tags: stringy --only url,domain binary

High-confidence strings: stringy --top 20 binary

Update "Example Output" section with actual output from test binaries

Remove "In Development" markers from CLI examples

Add note about error recovery and graceful degradation

Update "Development Status" to reflect completion of pipeline integration

This provides users with accurate, up-to-date information about the tool's capabilities.

Diagram

sequenceDiagram
    participant CLI as CLI (main.rs)
    participant Pipeline as Pipeline Orchestrator
    participant Container as Container Parser
    participant Extractor as String Extractor
    participant Classifier as Semantic Classifier
    participant Ranker as Ranking Engine
    participant Output as Output Formatter

    CLI->>Pipeline: run(binary_data, config)
    
    Pipeline->>Container: detect_format(data)
    Container-->>Pipeline: BinaryFormat
    
    Pipeline->>Container: create_parser(format)
    Container-->>Pipeline: Box<dyn ContainerParser>
    
    Pipeline->>Container: parse(data)
    alt Parse Success
        Container-->>Pipeline: ContainerInfo (sections, imports, exports)
    else Parse Error
        Container-->>Pipeline: Error
        Pipeline->>Pipeline: log_error(), try partial recovery
    end
    
    loop For each section with weight > threshold
        Pipeline->>Extractor: extract_from_section(data, section)
        alt Extraction Success
            Extractor-->>Pipeline: Vec<RawString>
        else Extraction Error
            Extractor-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), skip section
        end
    end
    
    Pipeline->>Pipeline: deduplicate_strings()
    
    loop For each extracted string
        Pipeline->>Classifier: classify(text, context)
        alt Classification Success
            Classifier-->>Pipeline: Vec<Tag>
        else Classification Error
            Classifier-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), mark unclassified
        end
        
        Pipeline->>Ranker: calculate_score(string, tags, section)
        alt Scoring Success
            Ranker-->>Pipeline: score (0-100)
        else Scoring Error
            Ranker-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), use default score
        end
    end
    
    Pipeline->>Pipeline: sort_by_score_descending()
    Pipeline->>Pipeline: apply_filters(config)
    
    Pipeline->>Output: format(strings, format_type)
    Output-->>Pipeline: formatted_output
    
    Pipeline-->>CLI: PipelineResult (strings, errors, warnings)
    CLI->>CLI: print output, handle errors

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

## Current State Analysis

The StringyMcStringFace project has a **solid foundation** with fully implemented container parsing for ELF, PE, and Mach-O formats. The container parsers provide comprehensive section classification, import/export extraction, and intelligent weighting systems. However, the **core pipeline components** (string extraction, classification, ranking, output) are **empty stubs**.

**What's Complete:**
- ✅ Container parsing with sophisticated section classification and weighting
- ✅ Import/export symbol extraction for all three formats
- ✅ Core data types (`FoundString`, `ContainerInfo`, `Tag`, `Encoding`, etc.)
- ✅ Error handling framework with `StringyError` enum
- ✅ Integration test infrastructure with binary fixtures
- ✅ CLI argument parsing structure (basic)

**What's Missing:**
- ❌ String extraction engines (ASCII/UTF-8, UTF-16)
- ❌ Semantic classification system (pattern matching, tagging)
- ❌ Ranking/scoring algorithm implementation
- ❌ Output formatters (JSON, human-readable, YARA)
- ❌ Pipeline orchestration module
- ❌ CLI integration with pipeline

**Critical Dependencies:**
The GitHub issue explicitly states this task depends on string extraction engines and semantic classification being functional. Since these don't exist yet, this plan must implement them **first** before the pipeline integration can occur.

**Architecture Insights:**
- The design documents specify a clear pipeline: Format Detection → Container Parsing → String Extraction → Classification → Ranking → Output
- Section weights (1.0-10.0) are already calculated by parsers
- The scoring formula is defined: `Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty`
- Test fixtures exist with known strings ("Hello, world!\n", "Helper called\n")

### Approach

This implementation follows a **bottom-up approach**, building the missing pipeline components before integrating them:

1. **String Extraction Layer** - Implement ASCII/UTF-8 and UTF-16 extraction engines with configurable parameters
2. **Classification Layer** - Build semantic pattern matching and symbol processing
3. **Ranking Layer** - Implement the scoring algorithm with section weights, semantic boosts, and noise penalties
4. **Output Layer** - Create formatters for JSON, human-readable, and YARA outputs
5. **Pipeline Orchestration** - Wire all components together with error recovery
6. **CLI Integration** - Connect the pipeline to command-line arguments
7. **Integration Testing** - Comprehensive end-to-end tests with error recovery scenarios

The approach prioritizes **graceful degradation** at each stage, allowing partial results even when individual components fail. Each module will be self-contained with clear interfaces, making the system testable and maintainable.

### Reasoning

I explored the repository structure and identified that container parsing is fully implemented while extraction, classification, ranking, and output modules are empty. I examined the ELF, PE, and Mach-O parsers to understand the section weighting system, reviewed the design documents to understand the intended architecture, checked the test fixtures to see what strings should be extracted, and analyzed the CLI documentation to understand the expected interface. The issue description provided the pipeline architecture and error recovery requirements.

## Mermaid Diagram

sequenceDiagram
    participant CLI as CLI (main.rs)
    participant Pipeline as Pipeline Orchestrator
    participant Container as Container Parser
    participant Extractor as String Extractor
    participant Classifier as Semantic Classifier
    participant Ranker as Ranking Engine
    participant Output as Output Formatter

    CLI->>Pipeline: run(binary_data, config)
    
    Pipeline->>Container: detect_format(data)
    Container-->>Pipeline: BinaryFormat
    
    Pipeline->>Container: create_parser(format)
    Container-->>Pipeline: Box<dyn ContainerParser>
    
    Pipeline->>Container: parse(data)
    alt Parse Success
        Container-->>Pipeline: ContainerInfo (sections, imports, exports)
    else Parse Error
        Container-->>Pipeline: Error
        Pipeline->>Pipeline: log_error(), try partial recovery
    end
    
    loop For each section with weight > threshold
        Pipeline->>Extractor: extract_from_section(data, section)
        alt Extraction Success
            Extractor-->>Pipeline: Vec<RawString>
        else Extraction Error
            Extractor-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), skip section
        end
    end
    
    Pipeline->>Pipeline: deduplicate_strings()
    
    loop For each extracted string
        Pipeline->>Classifier: classify(text, context)
        alt Classification Success
            Classifier-->>Pipeline: Vec<Tag>
        else Classification Error
            Classifier-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), mark unclassified
        end
        
        Pipeline->>Ranker: calculate_score(string, tags, section)
        alt Scoring Success
            Ranker-->>Pipeline: score (0-100)
        else Scoring Error
            Ranker-->>Pipeline: Error
            Pipeline->>Pipeline: log_warning(), use default score
        end
    end
    
    Pipeline->>Pipeline: sort_by_score_descending()
    Pipeline->>Pipeline: apply_filters(config)
    
    Pipeline->>Output: format(strings, format_type)
    Output-->>Pipeline: formatted_output
    
    Pipeline-->>CLI: PipelineResult (strings, errors, warnings)
    CLI->>CLI: print output, handle errors

## Proposed File Changes

### src/extraction/mod.rs(MODIFY)

References: 

- src/types.rs

Create the string extraction framework with traits and core types:

- Define `StringExtractor` trait with `extract_from_section()` method that takes section data and metadata, returns vector of extracted strings
- Define `RawString` struct to hold extracted string data before classification: text (String), offset (u64), encoding (Encoding), section_name (Option<String>), source (StringSource)
- Define `ExtractionConfig` struct with configurable parameters: min_ascii_length (default 4), min_wide_length (default 3), enabled_encodings (Vec<Encoding>), max_string_length (default 1024)
- Implement helper function `extract_strings_from_section()` that dispatches to appropriate extractor based on encoding configuration
- Add module declarations for `ascii` and `utf16` submodules

This provides the foundation for string extraction that will be used by the pipeline orchestrator.

### src/extraction/ascii.rs(NEW)

References: 

- src/extraction/mod.rs(MODIFY)
- src/types.rs

Implement ASCII/UTF-8 string extraction:

- Create `AsciiExtractor` struct with configurable minimum length
- Implement `StringExtractor` trait for `AsciiExtractor`
- Implement `extract_from_section()` to scan byte sequences for printable ASCII characters (0x20-0x7E, plus common extended characters like tabs and newlines)
- Extract null-terminated strings and sequences of printable characters
- Filter out obvious noise patterns: repeated padding bytes (0x00, 0xFF), table-like data with regular patterns
- Consider section context from `SectionInfo` to apply appropriate heuristics (e.g., more lenient in `.rodata` vs `.data`)
- Return `RawString` instances with proper offset, encoding (Ascii or Utf8), and section metadata
- Add unit tests for basic extraction, noise filtering, and edge cases (empty sections, all-noise data, mixed content)

This implements the primary string extraction mechanism for ASCII-encoded strings.

### src/extraction/utf16.rs(NEW)

References: 

- src/extraction/mod.rs(MODIFY)
- src/types.rs

Implement UTF-16 string extraction:

- Create `Utf16Extractor` struct with configurable minimum length and confidence threshold
- Implement `StringExtractor` trait for `Utf16Extractor`
- Implement `extract_from_section()` to detect UTF-16LE and UTF-16BE encoded strings
- For UTF-16LE: scan for even-length sequences where low bytes are mostly printable (0x20-0x7E) and high bytes are mostly zero
- For UTF-16BE: similar logic but with byte order reversed
- Calculate confidence score based on ratio of valid characters to total length
- Handle null-interleaved text patterns (ASCII with null bytes between characters)
- Filter low-confidence extractions to avoid false positives
- Return `RawString` instances with appropriate encoding (Utf16Le or Utf16Be) and confidence metadata
- Add unit tests for UTF-16LE extraction, UTF-16BE extraction, null-interleaved patterns, and confidence scoring

This enables extraction of wide-character strings common in Windows PE binaries.

### src/extraction/dedup.rs(NEW)

References: 

- src/extraction/mod.rs(MODIFY)
- src/types.rs

Implement string deduplication:

- Create `deduplicate_strings()` function that takes vector of `RawString` and returns deduplicated vector
- Use HashMap with string text as key to identify duplicates
- For duplicates, preserve the first occurrence's metadata (offset, section) but track all occurrences
- Implement canonicalization: trim whitespace, normalize line endings, but preserve original for output
- Maintain a list of all offsets where each unique string appears
- Add unit tests for basic deduplication, metadata preservation, and handling of strings that appear in multiple sections

This reduces noise in output while preserving provenance information.

### src/classification/mod.rs(MODIFY)

References: 

- src/types.rs

Create the classification framework:

- Define `Classifier` trait with `classify()` method that takes string text and context, returns vector of `Tag`
- Define `StringContext` struct with section information, source type, and encoding to provide context for classification
- Create `SemanticClassifier` struct that holds compiled regex patterns for URL, domain, IP, path, GUID, email, Base64, format string, and user agent detection
- Implement lazy initialization of regex patterns using `once_cell` or similar to cache compiled patterns
- Implement `classify_string()` function that applies all pattern matchers and returns matching tags
- Add helper function `classify_import_export()` that tags strings from import/export sources
- Add module declarations for `semantic` and `symbols` submodules
- Add unit tests for basic classification and context handling

This provides the foundation for semantic tagging of extracted strings.

### src/classification/semantic.rs(NEW)

References: 

- src/classification/mod.rs(MODIFY)
- src/types.rs

Implement semantic pattern matching:

- Create regex patterns for each semantic category:
  - URL: `https?://[^\s]+` with validation
  - Domain: `[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+`
  - IPv4: `\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b`
  - IPv6: standard IPv6 pattern with hex groups
  - File path (POSIX): `/[a-zA-Z0-9_/.\-]+`
  - File path (Windows): `[A-Z]:\\[^\s]+` or `\\\\[^\s]+`
  - Registry path: `HKEY_[A-Z_]+\\[^\s]+`
  - GUID: `\{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}`
  - Email: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
  - Base64: `[A-Za-z0-9+/]{20,}={0,2}` (minimum length to reduce false positives)
  - Format string: `%[0-9]*[sdxfp]` patterns
  - User agent: strings containing "Mozilla", "Chrome", "Safari", etc.
- Implement `PatternMatcher` struct that caches compiled regexes
- Implement `match_patterns()` function that applies all patterns and returns matching tags
- Add validation logic to reduce false positives (e.g., validate IPv4 octets are 0-255)
- Add unit tests for each pattern type with positive and negative cases

This implements the core semantic classification logic.

### src/classification/symbols.rs(NEW)

References: 

- src/classification/mod.rs(MODIFY)
- src/types.rs

Implement symbol processing:

- Add `rustc-demangle` dependency to Cargo.toml if not present
- Create `SymbolProcessor` struct for handling symbol names
- Implement `demangle_rust_symbol()` function using `rustc_demangle::demangle()`
- Implement `process_import_name()` that tags import symbols with `Tag::Import` and applies demangling if applicable
- Implement `process_export_name()` that tags export symbols with `Tag::Export` and applies demangling
- Handle common symbol prefixes (underscore for C symbols, mangled C++ names)
- Add unit tests for Rust symbol demangling, import/export tagging, and handling of various symbol formats

This enables proper handling of symbol names from import/export tables.

### src/classification/ranking.rs(NEW)

References: 

- src/types.rs

Implement the ranking and scoring algorithm:

- Create `RankingEngine` struct with configurable weights and penalties
- Define `RankingConfig` with section weight multipliers, tag boost values, and noise penalty thresholds
- Implement `calculate_score()` function that computes: `Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty`
- Section weight component: use the weight from `SectionInfo` (already calculated by parsers, 1.0-10.0) and multiply by 10 to get base score
- Encoding confidence: +10 for clean ASCII/UTF-8, +15 for high-confidence UTF-16, +5 for low-confidence
- Semantic boost: URL/GUID/registry (+50), paths/format strings (+30), imports/exports (+20), other tags (+10)
- Noise penalty: high entropy (-30), excessive length >200 chars (-20), repeated patterns (-10), obvious table data (-15)
- Implement `calculate_entropy()` helper to detect high-entropy strings
- Implement `detect_repeated_patterns()` to identify padding or table data
- Implement `is_table_data()` heuristic based on regular spacing and similar characters
- Ensure final score is clamped to reasonable range (0-100)
- Add unit tests for score calculation with various combinations of factors

This implements the core ranking algorithm to prioritize relevant strings.

### src/output/mod.rs(MODIFY)

References: 

- src/types.rs

Create the output formatting framework:

- Define `OutputFormatter` trait with `format()` method that takes vector of `FoundString` and returns formatted string
- Define `OutputFormat` enum with variants: Human, Json, Yara
- Define `OutputConfig` struct with format selection, color settings, truncation limits
- Implement `format_output()` function that dispatches to appropriate formatter based on config
- Add module declarations for `json`, `human`, and `yara` submodules
- Add helper functions for common formatting tasks (escaping, truncation, color codes)

This provides the foundation for multiple output formats.

### src/output/json.rs(NEW)

References: 

- src/output/mod.rs(MODIFY)
- src/types.rs

Implement JSON Lines output formatter:

- Create `JsonFormatter` struct implementing `OutputFormatter` trait
- Implement `format()` to serialize each `FoundString` as a JSON object on a single line
- Use `serde_json::to_string()` for serialization (FoundString already has Serialize derive)
- Ensure output is valid JSONL format (one JSON object per line, no trailing comma)
- Include all fields: text, encoding, offset, rva, section, length, tags, score, source
- Handle null values appropriately (rva and section can be None)
- Add unit tests for formatting single strings, multiple strings, and strings with various field combinations

This implements the machine-readable output format for pipeline integration.

### src/output/human.rs(NEW)

References: 

- src/output/mod.rs(MODIFY)
- src/types.rs

Implement human-readable table output formatter:

- Create `HumanFormatter` struct implementing `OutputFormatter` trait with color support configuration
- Implement `format()` to create aligned table with columns: Score, Offset, Section, Encoding, Tags, String
- Calculate column widths dynamically based on content
- Truncate long strings with "..." indicator (default 60 characters)
- Apply color coding based on score: green (>80), yellow (50-80), red (<50) using ANSI escape codes
- Respect NO_COLOR environment variable to disable colors
- Format offset as hexadecimal (0x prefix)
- Join multiple tags with commas
- Add table header with separator line
- Add unit tests for formatting, color coding, truncation, and column alignment

This implements the interactive output format for manual analysis.

### src/output/yara.rs(NEW)

References: 

- src/output/mod.rs(MODIFY)
- src/types.rs

Implement YARA rule output formatter:

- Create `YaraFormatter` struct implementing `OutputFormatter` trait
- Implement `format()` to generate valid YARA rule syntax
- Create rule header with metadata: description, generated_by, timestamp
- Group strings by semantic category (URLs, GUIDs, paths, etc.)
- Escape special characters in strings: backslash, quotes, newlines, tabs
- Add appropriate modifiers: `ascii` for ASCII strings, `wide` for UTF-16, `ascii wide` for both
- Convert binary data to hex format when necessary
- Add comments with score and classification information
- Generate simple condition: `any of them`
- Truncate very long strings (>100 chars) with comment indicating truncation
- Add unit tests for escaping, grouping, modifier selection, and complete rule generation

This implements the YARA-friendly output format for security rule creation.

### src/pipeline.rs(NEW)

References: 

- src/types.rs
- src/container/mod.rs
- src/extraction/mod.rs(MODIFY)
- src/classification/mod.rs(MODIFY)
- src/output/mod.rs(MODIFY)

Create the pipeline orchestration module:

- Define `Pipeline` struct that holds configuration and accumulated state
- Define `PipelineStage` enum with variants: FormatDetection, ContainerParsing, StringExtraction, Classification, Ranking, OutputFormatting
- Define `PipelineResult` struct that accumulates results and tracks errors: strings (Vec<FoundString>), errors (Vec<PipelineError>), warnings (Vec<String>), metadata (PipelineMetadata)
- Define `PipelineError` struct with stage (PipelineStage), error (StringyError), recovery_action (String)
- Define `PipelineMetadata` struct with format (BinaryFormat), sections_analyzed (Vec<String>), total_strings_found (usize), extraction_time_ms (u64)
- Define `PipelineConfig` struct with extraction_config (ExtractionConfig), ranking_config (RankingConfig), output_config (OutputConfig), max_errors (Option<usize>), strict_mode (bool)

- Implement `Pipeline::new()` constructor that takes config
- Implement `Pipeline::run()` main orchestration function that:
  1. Detects format using `container::detect_format()`
  2. Creates parser using `container::create_parser()`
  3. Parses container to get `ContainerInfo`
  4. Extracts strings from sections using extraction engines
  5. Adds import/export names as strings
  6. Deduplicates strings
  7. Classifies strings with semantic tags
  8. Calculates scores using ranking engine
  9. Sorts by score descending
  10. Formats output

- Implement error recovery at each stage:
  - Format detection failure: return error (cannot proceed without format)
  - Parser errors: log error, try to continue with partial section data
  - Extraction failures: log warning, skip problematic section, continue with others
  - Classification errors: log warning, mark string as unclassified, continue
  - Scoring failures: log warning, use default score (50), continue
  - Output formatting errors: return error (final stage, must succeed)

- Implement `Pipeline::log_error()` to track errors with stage context
- Implement `Pipeline::log_warning()` to track non-fatal issues
- Implement `Pipeline::should_continue()` to check if max_errors threshold reached
- Add comprehensive error messages with recovery actions taken

This implements the core pipeline orchestration with graceful degradation.

### src/lib.rs(MODIFY)

References: 

- src/pipeline.rs(NEW)

Update library exports to include new modules:

- Add `pub mod pipeline;` declaration
- Re-export pipeline types: `Pipeline`, `PipelineConfig`, `PipelineResult`, `PipelineStage`
- Update documentation to reflect that pipeline integration is now complete
- Update implementation status comments to mark extraction, classification, ranking, and output as complete

This makes the pipeline module accessible to external users of the library.

### src/main.rs(MODIFY)

References: 

- src/pipeline.rs(NEW)
- src/types.rs

Implement the CLI integration with pipeline:

- Expand `Cli` struct to include all command-line arguments:
  - `input: PathBuf` (already exists)
  - `min_len: Option<usize>` with default 4
  - `max_len: Option<usize>` with default 1024
  - `enc: Option<String>` for comma-separated encodings
  - `only_tags: Option<String>` for tag filtering
  - `exclude_tags: Option<String>` for tag exclusion
  - `sections: Option<String>` for section filtering
  - `top: Option<usize>` for result limiting (default 100)
  - `all: bool` flag to show all results
  - `format: Option<String>` with values: human, json, yara
  - `json: bool` shorthand flag
  - `yara: bool` shorthand flag
  - `max_errors: Option<usize>` for error threshold
  - `strict: bool` for fail-fast mode
  - `verbose: bool` for pipeline stage progress

- Implement `main()` function:
  1. Parse CLI arguments using clap
  2. Read binary file using `std::fs::read()`
  3. Build `PipelineConfig` from CLI arguments
  4. Create `Pipeline` instance with config
  5. Run pipeline with `pipeline.run(&data)`
  6. Handle `PipelineResult`: print warnings/errors if verbose, output formatted strings
  7. Return appropriate exit code based on errors

- Implement helper function `parse_encodings()` to convert comma-separated string to Vec<Encoding>
- Implement helper function `parse_tags()` to convert comma-separated string to Vec<Tag>
- Add error handling for file I/O errors with clear messages
- Add progress indicators if verbose flag is set

This connects the CLI to the pipeline orchestrator.

### Cargo.toml(MODIFY)

Add required dependencies for new functionality:

- Add `regex = "1.10"` for pattern matching in semantic classification
- Add `rustc-demangle = "0.1"` for Rust symbol demangling
- Add `once_cell = "1.19"` for lazy static regex compilation
- Consider adding `memmap2 = "0.9"` for memory-mapped file I/O (optional, can be added later for performance)

These dependencies are needed for the classification and symbol processing modules.

### tests/integration/pipeline_tests.rs(NEW)

References: 

- src/pipeline.rs(NEW)
- tests/fixtures/test_binary_elf
- tests/fixtures/test_binary_pe.exe
- tests/fixtures/test_binary_macho

Create comprehensive integration tests for the pipeline:

**Happy Path Tests:**
- `test_elf_complete_pipeline()`: Load `test_binary_elf` fixture, run full pipeline, verify strings extracted ("Hello, world!", "Helper called"), verify imports found (printf, malloc, free), verify exports found (main, exported_function, helper_function), verify scores assigned, verify output formats work
- `test_pe_complete_pipeline()`: Load `test_binary_pe.exe` fixture, run full pipeline, verify PE-specific handling, verify UTF-16 extraction if applicable, verify section classification
- `test_macho_complete_pipeline()`: Load `test_binary_macho` fixture, run full pipeline, verify Mach-O section handling, verify symbol extraction
- `test_json_output_format()`: Run pipeline with JSON output, parse output, validate JSON structure matches schema, verify all required fields present
- `test_human_output_format()`: Run pipeline with human-readable output, verify table formatting, verify column alignment
- `test_yara_output_format()`: Run pipeline with YARA output, verify valid YARA syntax, verify proper escaping
- `test_semantic_tagging()`: Create test binary with known URLs, paths, GUIDs, verify they are correctly tagged

**Error Recovery Tests:**
- `test_corrupted_section_recovery()`: Create binary with intentionally corrupted section data, verify pipeline continues with other sections, verify error logged but partial results returned
- `test_unknown_format_fallback()`: Provide non-binary data, verify graceful error handling with clear message
- `test_partial_utf16_recovery()`: Create data with incomplete UTF-16 sequences, verify extractor handles gracefully, continues with valid strings
- `test_empty_sections_handling()`: Create binary with zero-size sections, verify pipeline skips them without errors
- `test_max_errors_threshold()`: Configure pipeline with max_errors=3, trigger multiple errors, verify pipeline stops after threshold
- `test_strict_mode_failure()`: Enable strict mode, trigger error, verify pipeline fails immediately

**Performance Tests:**
- `test_large_binary_performance()`: Create or use large binary (10MB+), verify processing completes in reasonable time (<5 seconds), verify memory usage reasonable
- `test_many_strings_handling()`: Create binary with thousands of strings, verify deduplication works, verify ranking performance

**Configuration Tests:**
- `test_encoding_filtering()`: Configure pipeline to extract only ASCII, verify UTF-16 strings not extracted
- `test_section_filtering()`: Configure pipeline to scan only .rodata, verify other sections skipped
- `test_tag_filtering()`: Configure pipeline to show only URL tags, verify other tags filtered from output
- `test_top_n_limiting()`: Configure pipeline with top=10, verify only 10 highest-scoring strings returned

Use `insta` for snapshot testing where appropriate. Each test should verify both success conditions and proper error handling.

### tests/integration(NEW)

Create integration test directory to organize end-to-end tests separately from unit tests. This directory will contain `pipeline_tests.rs` and potentially other integration test files in the future.

### docs/src/architecture.md(MODIFY)

References: 

- src/pipeline.rs(NEW)

Update architecture documentation to reflect completed implementation:

- Change status indicators from 🚧 to ✅ for Extraction, Classification, Ranking, and Output modules
- Add section describing the Pipeline module with diagram showing data flow through stages
- Document error recovery strategy with examples of each recovery scenario
- Add section on PipelineConfig and how it controls behavior
- Update "Current Implementation Details" section to include extraction, classification, ranking, and output implementations
- Add examples of pipeline usage from code
- Update module dependency diagram to include pipeline.rs

This keeps documentation in sync with implementation.

### docs/src/troubleshooting.md(MODIFY)

References: 

- src/pipeline.rs(NEW)

Add troubleshooting guide for pipeline errors:

- Add section "Pipeline Error Recovery" explaining how the pipeline handles errors at each stage
- Document common error scenarios:
  - "Unsupported file format" - what to check, how to verify file type
  - "Section parsing failed" - how pipeline recovers, what partial results to expect
  - "String extraction errors" - encoding issues, how to adjust min-len
  - "Classification failures" - regex compilation errors, how to disable problematic patterns
  - "Output formatting errors" - invalid characters, escaping issues
- Add section "Debugging Pipeline Issues" with --verbose flag usage
- Add section "Performance Issues" with tips for large binaries
- Add examples of error messages and their meanings
- Add section "Strict Mode vs. Graceful Degradation" explaining when to use each

This helps users understand and resolve pipeline-related issues.

### README.md(MODIFY)

Update README to reflect completed implementation:

- Change "Current Status" section to mark all components as ✅ Complete
- Update "Usage" section with actual working examples using the implemented CLI flags
- Add "Quick Start" section with common use cases:
  - Basic analysis: `stringy binary`
  - JSON output: `stringy --json binary`
  - Filter by tags: `stringy --only url,domain binary`
  - High-confidence strings: `stringy --top 20 binary`
- Update "Example Output" section with actual output from test binaries
- Remove "In Development" markers from CLI examples
- Add note about error recovery and graceful degradation
- Update "Development Status" to reflect completion of pipeline integration

This provides users with accurate, up-to-date information about the tool's capabilities.

Developer Humor

🎭 The Pipeline's Tale

From chaos of bytes to meaning so clear,
Through stages of parsing, the strings appear.
When errors arise, we don't fall apart—
We log, we recover, we play it smart!

ASCII and UTF-16, side by side,
With tags and with scores, they're classified.
From ELF to PE to Mach-O we go,
Extracting the secrets the binaries know!

🔧 Error recovery: Because binaries are messy, but we're not!

Execution Information

Branch: main
Commit: bdd6206

💡 Tips

Supported Commands (Inside Comments)

Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Implemented: end-to-end pipeline with error recovery (catch_unwind in classification, typed errors, graceful raw-scan fallback) plus integration coverage. Closing as completed.

Complete End-to-End Pipeline Integration with Error Recovery and Testing #37

Description

Summary

Context

Pipeline Architecture

Proposed Solution

1. Pipeline Orchestration Module

2. Error Recovery Strategy

3. Data Flow Integration

4. End-to-End Integration Tests

Test Fixtures

Test Categories

5. Configuration & CLI Integration

Acceptance Criteria

Implementation Checklist

Dependencies

Related Issues

Task ID

Activity

unclesp1d3r commented on Nov 10, 2025

coderabbitai commented on Nov 10, 2025

✅ Title Enhanced

✅ Body Enhanced with:

1. Detailed Context

2. Visual Pipeline Architecture

3. Comprehensive Proposed Solution

4. Measurable Acceptance Criteria

5. Implementation Checklist

✅ Labels Updated

🔗 View Updated Issue

traycerai commented on Nov 10, 2025

Plan

Observations

Current State Analysis

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

unclesp1d3r commented on Jul 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions