Implement ASCII String Noise Filtering with Intelligent Heuristics

Summary

Implement intelligent heuristics to distinguish legitimate ASCII strings from binary noise, padding, and non-textual data during string extraction from binary files.

Problem Context

When extracting ASCII strings from binary files, the analyzer currently captures all byte sequences that technically qualify as ASCII characters. However, this leads to several quality issues:

Binary Noise: Random byte sequences that happen to fall within ASCII ranges but carry no semantic meaning
Padding Data: Repetitive null bytes, spaces, or other padding characters used for alignment
Table Data: Structured binary data (e.g., lookup tables, numerical arrays) that may contain ASCII-range values but aren't human-readable strings
Low Entropy Content: Sequences like repeated characters or patterns that don't represent meaningful text

Without proper filtering, these false positives pollute the extraction results, making it difficult for users to identify genuinely useful strings and significantly reducing the tool's practical value.

Proposed Solution

Implement a multi-layered heuristic filtering system in Rust:

1. Character Distribution Analysis

Calculate character frequency distributions
Flag strings with abnormal character ratios (e.g., >80% punctuation, all same character)
Apply entropy calculations to identify low-information content

2. Linguistic Patterns

Check for reasonable vowel-to-consonant ratios (for English-like strings)
Identify common word patterns and character bigrams/trigrams
Validate against common string patterns (paths, URLs, error messages)

3. Context-Aware Filtering

Leverage binary section information (.text, .data, .rodata, etc.)
Apply different heuristics based on section type
Consider string location and surrounding bytes

4. Configurable Thresholds

Allow users to adjust sensitivity levels
Provide options to include/exclude specific pattern types
Support allowlist/denylist patterns

5. Scoring System

Assign confidence scores to extracted strings
Allow filtering by minimum confidence threshold
Provide transparency into why strings were included/excluded

Acceptance Criteria

Technical Notes

Consider using the entropy crate for information theory calculations
Section context can be obtained from ELF/PE/Mach-O parsers (e.g., goblin crate)
Performance-critical code should be profiled and optimized
Heuristics should be modular to allow easy addition/removal

Requirements

1.4, 2.1

Dependencies

Blocked by: Basic ASCII String Extraction (Implement ASCII String Extractor with Configurable Length Filtering #9)

Task-ID

stringy-analyzer/ascii-noise-filtering

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

@unclesp1d3r

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the details of issue #10
gh issue view 10 --repo EvilBit-Labs/StringyMcStringFace --json title,body,labels,assignees,state,author

Length of output: 1280

🏁 Script executed:

#!/bin/bash
# Check available labels in the repository
gh label list --repo EvilBit-Labs/StringyMcStringFace --json name,description --limit 100

Length of output: 1182

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced content and metadata

# New enhanced body
cat > /tmp/issue_body.md << 'EOF'
## Summary

Implement intelligent heuristics to distinguish legitimate ASCII strings from binary noise, padding, and non-textual data during string extraction from binary files.

## Problem Context

When extracting ASCII strings from binary files, the analyzer currently captures all byte sequences that technically qualify as ASCII characters. However, this leads to several quality issues:

- **Binary Noise**: Random byte sequences that happen to fall within ASCII ranges but carry no semantic meaning
- **Padding Data**: Repetitive null bytes, spaces, or other padding characters used for alignment
- **Table Data**: Structured binary data (e.g., lookup tables, numerical arrays) that may contain ASCII-range values but aren't human-readable strings
- **Low Entropy Content**: Sequences like repeated characters or patterns that don't represent meaningful text

Without proper filtering, these false positives pollute the extraction results, making it difficult for users to identify genuinely useful strings and significantly reducing the tool's practical value.

## Proposed Solution

Implement a multi-layered heuristic filtering system in Rust:

### 1. **Character Distribution Analysis**
- Calculate character frequency distributions
- Flag strings with abnormal character ratios (e.g., >80% punctuation, all same character)
- Apply entropy calculations to identify low-information content

### 2. **Linguistic Patterns**
- Check for reasonable vowel-to-consonant ratios (for English-like strings)
- Identify common word patterns and character bigrams/trigrams
- Validate against common string patterns (paths, URLs, error messages)

### 3. **Context-Aware Filtering**
- Leverage binary section information (.text, .data, .rodata, etc.)
- Apply different heuristics based on section type
- Consider string location and surrounding bytes

### 4. **Configurable Thresholds**
- Allow users to adjust sensitivity levels
- Provide options to include/exclude specific pattern types
- Support allowlist/denylist patterns

### 5. **Scoring System**
- Assign confidence scores to extracted strings
- Allow filtering by minimum confidence threshold
- Provide transparency into why strings were included/excluded

## Acceptance Criteria

- [ ] Implement character distribution analysis functions
- [ ] Implement linguistic pattern detection
- [ ] Integrate section context into filtering decisions
- [ ] Add configurable threshold parameters
- [ ] Implement confidence scoring system
- [ ] Reduce false positive rate by >70% on test binaries
- [ ] Maintain >95% true positive retention
- [ ] Add comprehensive unit tests for each heuristic
- [ ] Document heuristic algorithms and tuning parameters
- [ ] Benchmark performance impact (<10% overhead target)

## Technical Notes

- Consider using the `entropy` crate for information theory calculations
- Section context can be obtained from ELF/PE/Mach-O parsers (e.g., `goblin` crate)
- Performance-critical code should be profiled and optimized
- Heuristics should be modular to allow easy addition/removal

## Requirements

1.4, 2.1

## Dependencies

- **Blocked by**: Basic ASCII String Extraction (#TBD)

## Task-ID

`stringy-analyzer/ascii-noise-filtering`
EOF

# Update the issue
gh issue edit 10 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --title "Implement ASCII String Noise Filtering with Intelligent Heuristics" \
  --body-file /tmp/issue_body.md \
  --add-label "priority:medium" \
  --add-label "MVP"

echo "Issue #10 updated successfully!"

Length of output: 94

@unclesp1d3r I've successfully enhanced issue #10! Here's what I updated:

Changes Made:

📝 Title

Before: "ASCII Noise Filtering"
After: "Implement ASCII String Noise Filtering with Intelligent Heuristics"
More descriptive and action-oriented

📄 Body Enhancements

Added comprehensive sections including:

Problem Context - Explains why noise filtering is critical, covering:
- Binary noise issues
- Padding data problems
- Table data false positives
- Low entropy content challenges
Proposed Solution - Multi-layered approach with:
- Character distribution analysis
- Linguistic pattern detection
- Context-aware filtering (section-based)
- Configurable thresholds
- Confidence scoring system
Acceptance Criteria - 10 concrete checkboxes including:
- Implementation requirements
- Performance targets (>70% false positive reduction, >95% true positive retention)
- Testing and documentation needs
- Performance overhead target (<10%)
Technical Notes - Implementation guidance:
- Suggested Rust crates (entropy, goblin)
- Performance considerations
- Modularity requirements

🏷️ Labels Added

priority:medium - Important for quality but not critical
MVP - Part of minimum viable product

You can view the updated issue here: #10

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

@unclesp1d3r

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the current state of issue #10 to see what enhancements have been applied
gh issue view 10 --repo EvilBit-Labs/StringyMcStringFace --json title,body,labels,assignees,state

Length of output: 4289

@unclesp1d3r Issue #10 has already been enhanced! Here's the current state:

✅ Current Status

📝 Title

"Implement ASCII String Noise Filtering with Intelligent Heuristics"

📄 Body Sections

The issue includes comprehensive sections:

Summary - Clear description of the feature
Problem Context - Binary noise, padding data, table data, and low entropy issues
Proposed Solution - 5-part multi-layered approach (character distribution, linguistic patterns, context-aware filtering, configurable thresholds, scoring system)
Acceptance Criteria - 10 concrete checkboxes with performance targets
Technical Notes - Implementation guidance with Rust crate suggestions
Requirements - 1.4, 2.1
Dependencies - Blocked by Basic ASCII String Extraction
Task-ID - stringy-analyzer/ascii-noise-filtering

🏷️ Labels Applied

area:analyzer - Binary analyzer functionality
type:enhancement - New feature request
status:backlog - In backlog
lang:rust - Rust implementation
needs:tests - Requires test coverage
MVP - Minimum viable product feature
priority:medium - Medium priority

👤 Assigned To

unclesp1d3r

The issue is fully enhanced and ready for development! If you'd like me to make any additional modifications or adjustments to the content, labels, or metadata, please let me know what changes you'd like.

@traycerai generate

@unclesp1d3r

Hi @unclesp1d3r, I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The codebase has a well-established architecture with complete container parsing (ELF, PE, Mach-O) and format-specific extraction modules (PE resources, Mach-O load commands). The FoundString type is fully defined with fields for text, encoding, offset, RVA, section, length, tags, score, and source. The classification module exists but is empty. No ASCII extraction or noise filtering implementation exists yet. The project uses goblin for binary parsing, pelite for PE resources, and has comprehensive testing with insta snapshots and criterion benchmarks.

Approach

Implement ASCII string noise filtering as a multi-layered system: (1) Create basic ASCII extraction with configurable minimum length, (2) Implement modular heuristic filters for character distribution, entropy, linguistic patterns, and context awareness, (3) Add a confidence scoring system that combines multiple heuristics, (4) Make thresholds configurable through a new configuration struct, (5) Integrate with existing section metadata for context-aware filtering. Use the entropy crate for Shannon entropy calculations and follow existing patterns from PE resource and Mach-O extraction modules.

Reasoning

Explored the project structure, examined existing extraction modules (PE resources, Mach-O load commands), reviewed the types and container parsers, checked the requirements and design documents, and researched available Rust entropy calculation crates.

Proposed File Changes
📄 Cargo.toml ^(MODIFY) 🔗
Add the entropy crate dependency (version 0.4 or latest) to enable Shannon entropy calculations for noise detection. This lightweight crate provides shannon_entropy function that takes byte slices and returns bits per byte, which will be used in the character distribution analysis heuristic.
Add the new benchmark to the Cargo.toml:
Add benchmark entry:
[[bench]]
name = "ascii_extraction"
harness = false
This follows the existing pattern for the elf and pe benchmarks already defined in the file.
📄 src/extraction/ascii.rs ^(NEW) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

📄 src/types.rs ^(MODIFY)

Create the ASCII string extraction module with the following components:

Basic Extraction Function (extract_ascii_strings):

Accept parameters: binary data slice, section info, and extraction config

Scan for contiguous runs of printable ASCII characters (0x20-0x7E)

Apply minimum length threshold from config (default 4 characters)

Return vector of candidate strings with their offsets and section context

String Candidate Structure (AsciiCandidate):

Store raw text, file offset, section reference, and byte length

Include preliminary metadata needed for filtering

Extraction Algorithm:

Iterate through section data byte-by-byte

Track current string being built

On printable character: append to current string

On non-printable: finalize current string if it meets minimum length

Handle null-terminated strings properly

Respect section boundaries

Integration Points:

Accept SectionInfo reference to access section type, weight, and permissions

Use ExtractionConfig for configurable parameters

Return candidates that will be filtered by the noise filtering module

Follow the pattern established in src/extraction/pe_resources.rs and src/extraction/macho_load_commands.rs for consistency.

📄 src/extraction/filters.rs ^(NEW) 🔗

References

📄 .kiro/specs/stringy-binary-analyzer/design.md

📄 src/types.rs ^(MODIFY)

Create the noise filtering module with modular heuristic filters:

Filter Trait (NoiseFilter):

Define trait with method calculate_confidence(text: &str, context: &FilterContext) -> f32

Return confidence score between 0.0 (definitely noise) and 1.0 (definitely legitimate)

Allow filters to be composed and weighted

FilterContext Structure:

Store section type, section name, surrounding bytes (for context)

Include section weight from container parser

Provide access to section permissions (executable, writable)

Character Distribution Filter (CharDistributionFilter):

Calculate character frequency distribution

Detect abnormal ratios: >80% punctuation, >90% same character, >70% non-alphanumeric

Check for reasonable mix of character types

Return low confidence for strings with extreme distributions

Entropy Filter (EntropyFilter):

Use entropy::shannon_entropy to calculate information content

Flag very low entropy (<1.5 bits/byte) as likely padding/repetition

Flag very high entropy (>7.5 bits/byte) as likely binary noise

Optimal range for text: 3.5-6.0 bits/byte

Make thresholds configurable

Linguistic Pattern Filter (LinguisticFilter):

Calculate vowel-to-consonant ratio (reasonable range: 0.2-0.8 for English)

Check for common bigrams (th, he, in, er, an, re, on, at, en, nd)

Detect word-like patterns (alternating consonants and vowels)

Handle non-English strings gracefully (don't penalize too heavily)

Length Filter (LengthFilter):

Penalize excessively long strings (>200 characters) as likely table data

Penalize very short strings in low-weight sections

Adjust thresholds based on section type

Repetition Filter (RepetitionFilter):

Detect repeated character patterns (e.g., "AAAA", "0000", "\x00\x00\x00\x00")

Identify repeated substrings (e.g., "abcabcabc")

Flag padding patterns (spaces, nulls, specific bytes)

Context-Aware Filter (ContextFilter):

Boost confidence for strings in high-weight sections (.rodata, .rdata, __cstring)

Reduce confidence for strings in code sections

Consider section permissions (read-only data more likely to contain strings)

Apply different thresholds based on section type

Composite Filter (CompositeNoiseFilter):

Combine multiple filters with configurable weights

Calculate weighted average of confidence scores

Provide overall confidence assessment

Allow enabling/disabling individual filters

Each filter should be independently testable and configurable. Reference the design document's noise detection requirements (Requirement 5.3) and the NoiseDetector trait pattern from src/classification/ranking.rs (mentioned in design but not yet implemented).

📄 src/extraction/config.rs ^(NEW) 🔗

References

📄 .kiro/specs/stringy-binary-analyzer/design.md

📄 src/types.rs ^(MODIFY)

Create the extraction configuration module:

ExtractionConfig Structure:

min_ascii_length: usize (default: 4) - minimum length for ASCII strings

min_wide_length: usize (default: 3) - minimum length for UTF-16 strings (for future use)

enabled_encodings: Vec<Encoding> - which encodings to extract

noise_filtering_enabled: bool (default: true) - enable/disable noise filtering

min_confidence_threshold: f32 (default: 0.5) - minimum confidence to include string

NoiseFilterConfig Structure:

entropy_min: f32 (default: 1.5) - minimum entropy threshold

entropy_max: f32 (default: 7.5) - maximum entropy threshold

max_length: usize (default: 200) - maximum string length before penalty

max_repetition_ratio: f32 (default: 0.7) - maximum ratio of repeated characters

min_vowel_ratio: f32 (default: 0.1) - minimum vowel ratio for linguistic filter

max_vowel_ratio: f32 (default: 0.9) - maximum vowel ratio for linguistic filter

filter_weights: FilterWeights - weights for combining filter scores

FilterWeights Structure:

entropy_weight: f32 (default: 0.25)

char_distribution_weight: f32 (default: 0.20)

linguistic_weight: f32 (default: 0.20)

length_weight: f32 (default: 0.15)

repetition_weight: f32 (default: 0.10)

context_weight: f32 (default: 0.10)

Ensure weights sum to 1.0

Default Implementation:

Provide sensible defaults based on requirements and design documents

Allow easy customization through builder pattern or direct field access

Validation:

Validate that thresholds are in reasonable ranges

Ensure filter weights sum to 1.0

Provide helpful error messages for invalid configurations

Reference the design document's ExtractionConfig structure and the requirements for configurable thresholds (Requirement 1.4, Acceptance Criteria 4).

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

Update the extraction module to include the new ASCII extraction and filtering modules:

Add module declarations:

pub mod ascii; - ASCII string extraction

pub mod filters; - noise filtering heuristics

pub mod config; - extraction configuration

Re-export key types and functions:

pub use ascii::extract_ascii_strings;

pub use config::{ExtractionConfig, NoiseFilterConfig, FilterWeights};

pub use filters::{NoiseFilter, CompositeNoiseFilter};

Update module documentation:

Document the new ASCII extraction and noise filtering capabilities

Provide usage examples showing how to extract ASCII strings with filtering

Explain the confidence scoring system

Reference the existing PE resource and Mach-O extraction examples

Maintain backward compatibility:

Keep existing exports for extract_resources, extract_resource_strings, and extract_load_command_strings

Ensure the module structure remains consistent with the existing pattern

📄 src/types.rs ^(MODIFY) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

Extend the types module to support noise filtering:

Add confidence field to FoundString structure:

pub confidence: f32 - confidence score from noise filtering (0.0-1.0)

This represents how confident we are that the string is legitimate vs noise

Default to 1.0 for strings from known-good sources (imports, exports, resources)

Set based on heuristic analysis for section-extracted strings

Update FoundString documentation:

Explain the confidence field and its interpretation

Document that confidence affects final scoring/ranking

Note that confidence is separate from the score field (which is for ranking)

Ensure backward compatibility:

Existing code that creates FoundString instances will need to set confidence

Update all existing FoundString creation sites in src/extraction/pe_resources.rs and src/extraction/macho_load_commands.rs to set confidence: 1.0 (since these are known-good sources)

Consider adding helper methods:

is_high_confidence(&self) -> bool - returns true if confidence >= 0.7

is_low_confidence(&self) -> bool - returns true if confidence < 0.5

Note: The score field is for final ranking (combining section weight, semantic boosts, and noise penalties), while confidence is specifically for the noise filtering assessment.

📄 src/extraction/pe_resources.rs ^(MODIFY) 🔗

References

📄 src/types.rs ^(MODIFY)

Update all FoundString creation sites to include the new confidence field:

In extract_version_info_strings function (around line 438):

Add confidence: 1.0 to the FoundString struct initialization

Version info strings are known-good, so they get maximum confidence

In extract_string_table_strings function (around line 593):

Add confidence: 1.0 to the FoundString struct initialization

String table resources are known-good sources

In manifest extraction (around line 780):

Add confidence: 1.0 to the FoundString struct initialization

Manifest strings are known-good sources

These changes ensure backward compatibility and maintain the semantic meaning that resource strings are high-confidence extractions.

📄 src/extraction/macho_load_commands.rs ^(MODIFY) 🔗

References

📄 src/types.rs ^(MODIFY)

Update all FoundString creation sites to include the new confidence field:

In extract_dylib_strings function (around line 101):

Add confidence: 1.0 to the FoundString struct initialization

Dylib paths from load commands are known-good sources

In extract_rpath_strings function (around line 129):

Add confidence: 1.0 to the FoundString struct initialization

Rpath strings from load commands are known-good sources

These changes maintain consistency with the PE resource extraction pattern and ensure that load command strings are marked as high-confidence.

📄 tests/test_ascii_extraction.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

Create comprehensive unit tests for ASCII extraction:

Basic Extraction Tests:

Test extraction of simple ASCII strings from byte arrays

Verify minimum length threshold is respected

Test null-terminated string handling

Test strings with mixed printable and non-printable characters

Edge Cases:

Empty input data

Data with no valid strings

Strings at section boundaries

Very long strings (>200 characters)

Single-character sequences

Section Context Tests:

Test extraction from different section types (.rodata, .data, .text)

Verify section metadata is properly attached to extracted strings

Test that section weights influence extraction decisions

Configuration Tests:

Test different minimum length thresholds

Test enabling/disabling noise filtering

Test custom confidence thresholds

Follow the testing patterns from tests/integration_pe.rs and use insta for snapshot testing where appropriate.

📄 tests/test_noise_filters.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

Create comprehensive unit tests for noise filtering heuristics:

Character Distribution Filter Tests:

Test detection of all-punctuation strings

Test detection of repeated character strings ("AAAA", "0000")

Test normal text strings (should pass)

Test mixed alphanumeric strings

Entropy Filter Tests:

Test low-entropy strings (padding, repetition): "\x00\x00\x00\x00", "AAAA"

Test high-entropy strings (random binary): random byte sequences

Test normal text entropy: "Hello, World!", "Error: file not found"

Test edge cases at threshold boundaries

Linguistic Filter Tests:

Test English-like strings with good vowel ratios: "Hello world", "Error message"

Test consonant-heavy strings: "bcdfghjklmnpqrstvwxyz"

Test vowel-heavy strings: "aeiouaeiou"

Test non-English strings (should not be overly penalized)

Test strings with numbers and symbols

Length Filter Tests:

Test very short strings (1-3 characters)

Test normal length strings (4-50 characters)

Test long strings (50-200 characters)

Test excessively long strings (>200 characters)

Repetition Filter Tests:

Test repeated characters: "AAAA", "0000", " "

Test repeated patterns: "abcabcabc", "123123123"

Test normal strings with some repetition: "Hello", "Mississippi"

Context Filter Tests:

Test strings from high-weight sections (.rodata) - should boost confidence

Test strings from low-weight sections (.text) - should reduce confidence

Test strings from writable sections - should apply appropriate adjustments

Composite Filter Tests:

Test combining multiple filters with default weights

Test custom filter weights

Test enabling/disabling individual filters

Test overall confidence calculation

Real-World Scenarios:

Test legitimate strings that should pass all filters

Test obvious noise that should fail multiple filters

Test edge cases that might pass some filters but fail others

Test strings from actual binary samples

Use test fixtures with known-good and known-bad strings. Include examples from the issue description (binary noise, padding data, table data, low entropy content).

📄 tests/test_ascii_integration.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

📄 tests/integration_elf.rs

Create integration tests for end-to-end ASCII extraction with noise filtering:

Binary Fixture Tests:

Use existing test binaries (test_binary_elf, test_binary_pe.exe, test_binary_macho)

Extract ASCII strings from each binary format

Verify that legitimate strings are extracted with high confidence

Verify that noise is filtered out or marked with low confidence

False Positive Reduction Tests:

Create test data with known noise patterns

Verify that noise is correctly identified and filtered

Measure false positive rate (target: >70% reduction per acceptance criteria)

True Positive Retention Tests:

Create test data with known legitimate strings

Verify that legitimate strings are retained

Measure true positive retention rate (target: >95% per acceptance criteria)

Performance Tests:

Measure extraction time with and without noise filtering

Verify that overhead is <10% (per acceptance criteria)

Test with various binary sizes

Snapshot Tests:

Use insta to create snapshots of extracted strings

Verify consistency across runs

Document expected output for regression testing

Follow the pattern from tests/integration_pe.rs for fixture handling and snapshot testing.

📄 benches/ascii_extraction.rs ^(NEW) 🔗

References

📄 benches/pe.rs

Create performance benchmarks for ASCII extraction and noise filtering:

Basic Extraction Benchmark:

Benchmark ASCII string extraction without filtering

Use test binary fixtures of various sizes

Measure throughput (bytes/second)

Filtered Extraction Benchmark:

Benchmark ASCII string extraction with full noise filtering

Compare against baseline extraction time

Verify <10% overhead target

Individual Filter Benchmarks:

Benchmark each filter independently

Identify performance bottlenecks

Measure filter execution time per string

Composite Filter Benchmark:

Benchmark the combined filter with all heuristics enabled

Test with different filter weight configurations

Measure impact of enabling/disabling individual filters

Entropy Calculation Benchmark:

Benchmark the entropy::shannon_entropy function

Test with strings of various lengths

Verify it's not a performance bottleneck

Large Binary Benchmark:

Test with large binary files (10MB, 50MB, 100MB)

Measure memory usage

Verify scalability

Follow the pattern from benches/pe.rs using criterion for benchmarking. Use black_box to prevent compiler optimizations from skewing results.

📄 docs/src/string-extraction.md ^(MODIFY) 🔗

Update the string extraction documentation to include ASCII noise filtering:

Add ASCII Extraction Section:

Document the ASCII string extraction process

Explain the minimum length threshold

Describe how strings are identified in binary data

Add Noise Filtering Section:

Document the multi-layered heuristic filtering system

Explain each filter type and its purpose:

Character distribution analysis

Entropy-based filtering

Linguistic pattern detection

Length-based filtering

Repetition detection

Context-aware filtering

Provide examples of strings that pass vs fail each filter

Add Configuration Section:

Document the ExtractionConfig and NoiseFilterConfig structures

Explain configurable thresholds and their defaults

Provide examples of customizing filter behavior

Document filter weights and how to adjust them

Add Confidence Scoring Section:

Explain the confidence score (0.0-1.0)

Describe how confidence is calculated from multiple heuristics

Document the relationship between confidence and final ranking

Provide guidance on interpreting confidence scores

Add Performance Section:

Document the performance characteristics of noise filtering

Explain the <10% overhead target

Provide tips for optimizing extraction performance

Add Examples:

Show code examples of using ASCII extraction with filtering

Demonstrate customizing filter configuration

Show how to filter results by confidence threshold

Reference the existing documentation structure and maintain consistency with the style used in other documentation files.

Diagram
sequenceDiagram
    participant User
    participant Extractor as ASCII Extractor
    participant Filters as Noise Filters
    participant Config as Configuration
    participant Section as Section Info
    
    User->>Config: Create ExtractionConfig
    Config-->>User: Config with thresholds
    
    User->>Extractor: extract_ascii_strings(data, section, config)
    Extractor->>Section: Get section type & weight
    Section-->>Extractor: Section metadata
    
    loop For each byte sequence
        Extractor->>Extractor: Scan for printable ASCII
        Extractor->>Extractor: Check minimum length
        alt String meets criteria
            Extractor->>Filters: Apply noise filters
            Filters->>Filters: Character distribution
            Filters->>Filters: Entropy calculation
            Filters->>Filters: Linguistic patterns
            Filters->>Filters: Length check
            Filters->>Filters: Repetition detection
            Filters->>Filters: Context awareness
            Filters-->>Extractor: Confidence score (0.0-1.0)
            
            alt Confidence >= threshold
                Extractor->>Extractor: Create FoundString
                Note over Extractor: Include confidence,<br/>section, offset, tags
            else Confidence < threshold
                Extractor->>Extractor: Discard as noise
            end
        end
    end
    
    Extractor-->>User: Vec<FoundString>
Loading
Import In IDE
🤖 Prompt for AI Agents
I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established architecture with complete container parsing (ELF, PE, Mach-O) and format-specific extraction modules (PE resources, Mach-O load commands). The `FoundString` type is fully defined with fields for text, encoding, offset, RVA, section, length, tags, score, and source. The classification module exists but is empty. No ASCII extraction or noise filtering implementation exists yet. The project uses `goblin` for binary parsing, `pelite` for PE resources, and has comprehensive testing with `insta` snapshots and `criterion` benchmarks.

### Approach

Implement ASCII string noise filtering as a multi-layered system: (1) Create basic ASCII extraction with configurable minimum length, (2) Implement modular heuristic filters for character distribution, entropy, linguistic patterns, and context awareness, (3) Add a confidence scoring system that combines multiple heuristics, (4) Make thresholds configurable through a new configuration struct, (5) Integrate with existing section metadata for context-aware filtering. Use the `entropy` crate for Shannon entropy calculations and follow existing patterns from PE resource and Mach-O extraction modules.

### Reasoning

Explored the project structure, examined existing extraction modules (PE resources, Mach-O load commands), reviewed the types and container parsers, checked the requirements and design documents, and researched available Rust entropy calculation crates.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant Extractor as ASCII Extractor
    participant Filters as Noise Filters
    participant Config as Configuration
    participant Section as Section Info
    
    User->>Config: Create ExtractionConfig
    Config-->>User: Config with thresholds
    
    User->>Extractor: extract_ascii_strings(data, section, config)
    Extractor->>Section: Get section type & weight
    Section-->>Extractor: Section metadata
    
    loop For each byte sequence
        Extractor->>Extractor: Scan for printable ASCII
        Extractor->>Extractor: Check minimum length
        alt String meets criteria
            Extractor->>Filters: Apply noise filters
            Filters->>Filters: Character distribution
            Filters->>Filters: Entropy calculation
            Filters->>Filters: Linguistic patterns
            Filters->>Filters: Length check
            Filters->>Filters: Repetition detection
            Filters->>Filters: Context awareness
            Filters-->>Extractor: Confidence score (0.0-1.0)
            
            alt Confidence >= threshold
                Extractor->>Extractor: Create FoundString
                Note over Extractor: Include confidence,<br/>section, offset, tags
            else Confidence < threshold
                Extractor->>Extractor: Discard as noise
            end
        end
    end
    
    Extractor-->>User: Vec<FoundString>

## Proposed File Changes

### Cargo.toml(MODIFY)

Add the `entropy` crate dependency (version 0.4 or latest) to enable Shannon entropy calculations for noise detection. This lightweight crate provides `shannon_entropy` function that takes byte slices and returns bits per byte, which will be used in the character distribution analysis heuristic.
Add the new benchmark to the Cargo.toml:

1. Add benchmark entry:
   ```toml
   [[bench]]
   name = "ascii_extraction"
   harness = false
   ```

This follows the existing pattern for the `elf` and `pe` benchmarks already defined in the file.

### src/extraction/ascii.rs(NEW)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)
- src/types.rs(MODIFY)

Create the ASCII string extraction module with the following components:

1. **Basic Extraction Function** (`extract_ascii_strings`):
   - Accept parameters: binary data slice, section info, and extraction config
   - Scan for contiguous runs of printable ASCII characters (0x20-0x7E)
   - Apply minimum length threshold from config (default 4 characters)
   - Return vector of candidate strings with their offsets and section context

2. **String Candidate Structure** (`AsciiCandidate`):
   - Store raw text, file offset, section reference, and byte length
   - Include preliminary metadata needed for filtering

3. **Extraction Algorithm**:
   - Iterate through section data byte-by-byte
   - Track current string being built
   - On printable character: append to current string
   - On non-printable: finalize current string if it meets minimum length
   - Handle null-terminated strings properly
   - Respect section boundaries

4. **Integration Points**:
   - Accept `SectionInfo` reference to access section type, weight, and permissions
   - Use `ExtractionConfig` for configurable parameters
   - Return candidates that will be filtered by the noise filtering module

Follow the pattern established in `src/extraction/pe_resources.rs` and `src/extraction/macho_load_commands.rs` for consistency.

### src/extraction/filters.rs(NEW)

References: 

- .kiro/specs/stringy-binary-analyzer/design.md
- src/types.rs(MODIFY)

Create the noise filtering module with modular heuristic filters:

1. **Filter Trait** (`NoiseFilter`):
   - Define trait with method `calculate_confidence(text: &str, context: &FilterContext) -> f32`
   - Return confidence score between 0.0 (definitely noise) and 1.0 (definitely legitimate)
   - Allow filters to be composed and weighted

2. **FilterContext Structure**:
   - Store section type, section name, surrounding bytes (for context)
   - Include section weight from container parser
   - Provide access to section permissions (executable, writable)

3. **Character Distribution Filter** (`CharDistributionFilter`):
   - Calculate character frequency distribution
   - Detect abnormal ratios: >80% punctuation, >90% same character, >70% non-alphanumeric
   - Check for reasonable mix of character types
   - Return low confidence for strings with extreme distributions

4. **Entropy Filter** (`EntropyFilter`):
   - Use `entropy::shannon_entropy` to calculate information content
   - Flag very low entropy (<1.5 bits/byte) as likely padding/repetition
   - Flag very high entropy (>7.5 bits/byte) as likely binary noise
   - Optimal range for text: 3.5-6.0 bits/byte
   - Make thresholds configurable

5. **Linguistic Pattern Filter** (`LinguisticFilter`):
   - Calculate vowel-to-consonant ratio (reasonable range: 0.2-0.8 for English)
   - Check for common bigrams (th, he, in, er, an, re, on, at, en, nd)
   - Detect word-like patterns (alternating consonants and vowels)
   - Handle non-English strings gracefully (don't penalize too heavily)

6. **Length Filter** (`LengthFilter`):
   - Penalize excessively long strings (>200 characters) as likely table data
   - Penalize very short strings in low-weight sections
   - Adjust thresholds based on section type

7. **Repetition Filter** (`RepetitionFilter`):
   - Detect repeated character patterns (e.g., "AAAA", "0000", "\x00\x00\x00\x00")
   - Identify repeated substrings (e.g., "abcabcabc")
   - Flag padding patterns (spaces, nulls, specific bytes)

8. **Context-Aware Filter** (`ContextFilter`):
   - Boost confidence for strings in high-weight sections (.rodata, .rdata, __cstring)
   - Reduce confidence for strings in code sections
   - Consider section permissions (read-only data more likely to contain strings)
   - Apply different thresholds based on section type

9. **Composite Filter** (`CompositeNoiseFilter`):
   - Combine multiple filters with configurable weights
   - Calculate weighted average of confidence scores
   - Provide overall confidence assessment
   - Allow enabling/disabling individual filters

Each filter should be independently testable and configurable. Reference the design document's noise detection requirements (Requirement 5.3) and the NoiseDetector trait pattern from `src/classification/ranking.rs` (mentioned in design but not yet implemented).

### src/extraction/config.rs(NEW)

References: 

- .kiro/specs/stringy-binary-analyzer/design.md
- src/types.rs(MODIFY)

Create the extraction configuration module:

1. **ExtractionConfig Structure**:
   - `min_ascii_length: usize` (default: 4) - minimum length for ASCII strings
   - `min_wide_length: usize` (default: 3) - minimum length for UTF-16 strings (for future use)
   - `enabled_encodings: Vec<Encoding>` - which encodings to extract
   - `noise_filtering_enabled: bool` (default: true) - enable/disable noise filtering
   - `min_confidence_threshold: f32` (default: 0.5) - minimum confidence to include string

2. **NoiseFilterConfig Structure**:
   - `entropy_min: f32` (default: 1.5) - minimum entropy threshold
   - `entropy_max: f32` (default: 7.5) - maximum entropy threshold
   - `max_length: usize` (default: 200) - maximum string length before penalty
   - `max_repetition_ratio: f32` (default: 0.7) - maximum ratio of repeated characters
   - `min_vowel_ratio: f32` (default: 0.1) - minimum vowel ratio for linguistic filter
   - `max_vowel_ratio: f32` (default: 0.9) - maximum vowel ratio for linguistic filter
   - `filter_weights: FilterWeights` - weights for combining filter scores

3. **FilterWeights Structure**:
   - `entropy_weight: f32` (default: 0.25)
   - `char_distribution_weight: f32` (default: 0.20)
   - `linguistic_weight: f32` (default: 0.20)
   - `length_weight: f32` (default: 0.15)
   - `repetition_weight: f32` (default: 0.10)
   - `context_weight: f32` (default: 0.10)
   - Ensure weights sum to 1.0

4. **Default Implementation**:
   - Provide sensible defaults based on requirements and design documents
   - Allow easy customization through builder pattern or direct field access

5. **Validation**:
   - Validate that thresholds are in reasonable ranges
   - Ensure filter weights sum to 1.0
   - Provide helpful error messages for invalid configurations

Reference the design document's ExtractionConfig structure and the requirements for configurable thresholds (Requirement 1.4, Acceptance Criteria 4).

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)

Update the extraction module to include the new ASCII extraction and filtering modules:

1. Add module declarations:
   - `pub mod ascii;` - ASCII string extraction
   - `pub mod filters;` - noise filtering heuristics
   - `pub mod config;` - extraction configuration

2. Re-export key types and functions:
   - `pub use ascii::extract_ascii_strings;`
   - `pub use config::{ExtractionConfig, NoiseFilterConfig, FilterWeights};`
   - `pub use filters::{NoiseFilter, CompositeNoiseFilter};`

3. Update module documentation:
   - Document the new ASCII extraction and noise filtering capabilities
   - Provide usage examples showing how to extract ASCII strings with filtering
   - Explain the confidence scoring system
   - Reference the existing PE resource and Mach-O extraction examples

4. Maintain backward compatibility:
   - Keep existing exports for `extract_resources`, `extract_resource_strings`, and `extract_load_command_strings`
   - Ensure the module structure remains consistent with the existing pattern

### src/types.rs(MODIFY)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)

Extend the types module to support noise filtering:

1. Add `confidence` field to `FoundString` structure:
   - `pub confidence: f32` - confidence score from noise filtering (0.0-1.0)
   - This represents how confident we are that the string is legitimate vs noise
   - Default to 1.0 for strings from known-good sources (imports, exports, resources)
   - Set based on heuristic analysis for section-extracted strings

2. Update `FoundString` documentation:
   - Explain the confidence field and its interpretation
   - Document that confidence affects final scoring/ranking
   - Note that confidence is separate from the score field (which is for ranking)

3. Ensure backward compatibility:
   - Existing code that creates `FoundString` instances will need to set confidence
   - Update all existing `FoundString` creation sites in `src/extraction/pe_resources.rs` and `src/extraction/macho_load_commands.rs` to set `confidence: 1.0` (since these are known-good sources)

4. Consider adding helper methods:
   - `is_high_confidence(&self) -> bool` - returns true if confidence >= 0.7
   - `is_low_confidence(&self) -> bool` - returns true if confidence < 0.5

Note: The `score` field is for final ranking (combining section weight, semantic boosts, and noise penalties), while `confidence` is specifically for the noise filtering assessment.

### src/extraction/pe_resources.rs(MODIFY)

References: 

- src/types.rs(MODIFY)

Update all `FoundString` creation sites to include the new `confidence` field:

1. In `extract_version_info_strings` function (around line 438):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Version info strings are known-good, so they get maximum confidence

2. In `extract_string_table_strings` function (around line 593):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - String table resources are known-good sources

3. In manifest extraction (around line 780):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Manifest strings are known-good sources

These changes ensure backward compatibility and maintain the semantic meaning that resource strings are high-confidence extractions.

### src/extraction/macho_load_commands.rs(MODIFY)

References: 

- src/types.rs(MODIFY)

Update all `FoundString` creation sites to include the new `confidence` field:

1. In `extract_dylib_strings` function (around line 101):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Dylib paths from load commands are known-good sources

2. In `extract_rpath_strings` function (around line 129):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Rpath strings from load commands are known-good sources

These changes maintain consistency with the PE resource extraction pattern and ensure that load command strings are marked as high-confidence.

### tests/test_ascii_extraction.rs(NEW)

References: 

- tests/integration_pe.rs

Create comprehensive unit tests for ASCII extraction:

1. **Basic Extraction Tests**:
   - Test extraction of simple ASCII strings from byte arrays
   - Verify minimum length threshold is respected
   - Test null-terminated string handling
   - Test strings with mixed printable and non-printable characters

2. **Edge Cases**:
   - Empty input data
   - Data with no valid strings
   - Strings at section boundaries
   - Very long strings (>200 characters)
   - Single-character sequences

3. **Section Context Tests**:
   - Test extraction from different section types (.rodata, .data, .text)
   - Verify section metadata is properly attached to extracted strings
   - Test that section weights influence extraction decisions

4. **Configuration Tests**:
   - Test different minimum length thresholds
   - Test enabling/disabling noise filtering
   - Test custom confidence thresholds

Follow the testing patterns from `tests/integration_pe.rs` and use `insta` for snapshot testing where appropriate.

### tests/test_noise_filters.rs(NEW)

References: 

- tests/integration_pe.rs

Create comprehensive unit tests for noise filtering heuristics:

1. **Character Distribution Filter Tests**:
   - Test detection of all-punctuation strings
   - Test detection of repeated character strings ("AAAA", "0000")
   - Test normal text strings (should pass)
   - Test mixed alphanumeric strings

2. **Entropy Filter Tests**:
   - Test low-entropy strings (padding, repetition): "\x00\x00\x00\x00", "AAAA"
   - Test high-entropy strings (random binary): random byte sequences
   - Test normal text entropy: "Hello, World!", "Error: file not found"
   - Test edge cases at threshold boundaries

3. **Linguistic Filter Tests**:
   - Test English-like strings with good vowel ratios: "Hello world", "Error message"
   - Test consonant-heavy strings: "bcdfghjklmnpqrstvwxyz"
   - Test vowel-heavy strings: "aeiouaeiou"
   - Test non-English strings (should not be overly penalized)
   - Test strings with numbers and symbols

4. **Length Filter Tests**:
   - Test very short strings (1-3 characters)
   - Test normal length strings (4-50 characters)
   - Test long strings (50-200 characters)
   - Test excessively long strings (>200 characters)

5. **Repetition Filter Tests**:
   - Test repeated characters: "AAAA", "0000", "    "
   - Test repeated patterns: "abcabcabc", "123123123"
   - Test normal strings with some repetition: "Hello", "Mississippi"

6. **Context Filter Tests**:
   - Test strings from high-weight sections (.rodata) - should boost confidence
   - Test strings from low-weight sections (.text) - should reduce confidence
   - Test strings from writable sections - should apply appropriate adjustments

7. **Composite Filter Tests**:
   - Test combining multiple filters with default weights
   - Test custom filter weights
   - Test enabling/disabling individual filters
   - Test overall confidence calculation

8. **Real-World Scenarios**:
   - Test legitimate strings that should pass all filters
   - Test obvious noise that should fail multiple filters
   - Test edge cases that might pass some filters but fail others
   - Test strings from actual binary samples

Use test fixtures with known-good and known-bad strings. Include examples from the issue description (binary noise, padding data, table data, low entropy content).

### tests/test_ascii_integration.rs(NEW)

References: 

- tests/integration_pe.rs
- tests/integration_elf.rs

Create integration tests for end-to-end ASCII extraction with noise filtering:

1. **Binary Fixture Tests**:
   - Use existing test binaries (test_binary_elf, test_binary_pe.exe, test_binary_macho)
   - Extract ASCII strings from each binary format
   - Verify that legitimate strings are extracted with high confidence
   - Verify that noise is filtered out or marked with low confidence

2. **False Positive Reduction Tests**:
   - Create test data with known noise patterns
   - Verify that noise is correctly identified and filtered
   - Measure false positive rate (target: >70% reduction per acceptance criteria)

3. **True Positive Retention Tests**:
   - Create test data with known legitimate strings
   - Verify that legitimate strings are retained
   - Measure true positive retention rate (target: >95% per acceptance criteria)

4. **Performance Tests**:
   - Measure extraction time with and without noise filtering
   - Verify that overhead is <10% (per acceptance criteria)
   - Test with various binary sizes

5. **Snapshot Tests**:
   - Use `insta` to create snapshots of extracted strings
   - Verify consistency across runs
   - Document expected output for regression testing

Follow the pattern from `tests/integration_pe.rs` for fixture handling and snapshot testing.

### benches/ascii_extraction.rs(NEW)

References: 

- benches/pe.rs

Create performance benchmarks for ASCII extraction and noise filtering:

1. **Basic Extraction Benchmark**:
   - Benchmark ASCII string extraction without filtering
   - Use test binary fixtures of various sizes
   - Measure throughput (bytes/second)

2. **Filtered Extraction Benchmark**:
   - Benchmark ASCII string extraction with full noise filtering
   - Compare against baseline extraction time
   - Verify <10% overhead target

3. **Individual Filter Benchmarks**:
   - Benchmark each filter independently
   - Identify performance bottlenecks
   - Measure filter execution time per string

4. **Composite Filter Benchmark**:
   - Benchmark the combined filter with all heuristics enabled
   - Test with different filter weight configurations
   - Measure impact of enabling/disabling individual filters

5. **Entropy Calculation Benchmark**:
   - Benchmark the `entropy::shannon_entropy` function
   - Test with strings of various lengths
   - Verify it's not a performance bottleneck

6. **Large Binary Benchmark**:
   - Test with large binary files (10MB, 50MB, 100MB)
   - Measure memory usage
   - Verify scalability

Follow the pattern from `benches/pe.rs` using `criterion` for benchmarking. Use `black_box` to prevent compiler optimizations from skewing results.

### docs/src/string-extraction.md(MODIFY)

Update the string extraction documentation to include ASCII noise filtering:

1. **Add ASCII Extraction Section**:
   - Document the ASCII string extraction process
   - Explain the minimum length threshold
   - Describe how strings are identified in binary data

2. **Add Noise Filtering Section**:
   - Document the multi-layered heuristic filtering system
   - Explain each filter type and its purpose:
     - Character distribution analysis
     - Entropy-based filtering
     - Linguistic pattern detection
     - Length-based filtering
     - Repetition detection
     - Context-aware filtering
   - Provide examples of strings that pass vs fail each filter

3. **Add Configuration Section**:
   - Document the `ExtractionConfig` and `NoiseFilterConfig` structures
   - Explain configurable thresholds and their defaults
   - Provide examples of customizing filter behavior
   - Document filter weights and how to adjust them

4. **Add Confidence Scoring Section**:
   - Explain the confidence score (0.0-1.0)
   - Describe how confidence is calculated from multiple heuristics
   - Document the relationship between confidence and final ranking
   - Provide guidance on interpreting confidence scores

5. **Add Performance Section**:
   - Document the performance characteristics of noise filtering
   - Explain the <10% overhead target
   - Provide tips for optimizing extraction performance

6. **Add Examples**:
   - Show code examples of using ASCII extraction with filtering
   - Demonstrate customizing filter configuration
   - Show how to filter results by confidence threshold

Reference the existing documentation structure and maintain consistency with the style used in other documentation files.
Developer Humor

🎯 The String Detective's Creed\n>\n> In a world of binary noise and padding bytes,\n> Where garbage strings give analysts the frights,\n> We filter with entropy, linguistics, and care,\n> To find the real strings hiding in there!\n>\n> No more \AAAA\ or random hex debris,\n> Just meaningful text, as clean as can be! 🔍✨

Execution Information

Branch: main
Commit: baf4526

Plan

Observations

The codebase has a well-established architecture with complete container parsing (ELF, PE, Mach-O) and format-specific extraction modules (PE resources, Mach-O load commands). The FoundString type is fully defined with fields for text, encoding, offset, RVA, section, length, tags, score, and source. The classification module exists but is empty. No ASCII extraction or noise filtering implementation exists yet. The project uses goblin for binary parsing, pelite for PE resources, and has comprehensive testing with insta snapshots and criterion benchmarks.

Approach

Implement ASCII string noise filtering as a multi-layered system: (1) Create basic ASCII extraction with configurable minimum length, (2) Implement modular heuristic filters for character distribution, entropy, linguistic patterns, and context awareness, (3) Add a confidence scoring system that combines multiple heuristics, (4) Make thresholds configurable through a new configuration struct, (5) Integrate with existing section metadata for context-aware filtering. Use the entropy crate for Shannon entropy calculations and follow existing patterns from PE resource and Mach-O extraction modules.

Reasoning

Explored the project structure, examined existing extraction modules (PE resources, Mach-O load commands), reviewed the types and container parsers, checked the requirements and design documents, and researched available Rust entropy calculation crates.

Proposed File Changes

📄 Cargo.toml ^(MODIFY) 🔗

Add the entropy crate dependency (version 0.4 or latest) to enable Shannon entropy calculations for noise detection. This lightweight crate provides shannon_entropy function that takes byte slices and returns bits per byte, which will be used in the character distribution analysis heuristic.
Add the new benchmark to the Cargo.toml:
Add benchmark entry:
[[bench]]
name = "ascii_extraction"
harness = false
This follows the existing pattern for the elf and pe benchmarks already defined in the file.

📄 src/extraction/ascii.rs ^(NEW) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

📄 src/types.rs ^(MODIFY)

Create the ASCII string extraction module with the following components:

Basic Extraction Function (extract_ascii_strings):

Accept parameters: binary data slice, section info, and extraction config

Scan for contiguous runs of printable ASCII characters (0x20-0x7E)

Apply minimum length threshold from config (default 4 characters)

Return vector of candidate strings with their offsets and section context

String Candidate Structure (AsciiCandidate):

Store raw text, file offset, section reference, and byte length

Include preliminary metadata needed for filtering

Extraction Algorithm:

Iterate through section data byte-by-byte

Track current string being built

On printable character: append to current string

On non-printable: finalize current string if it meets minimum length

Handle null-terminated strings properly

Respect section boundaries

Integration Points:

Accept SectionInfo reference to access section type, weight, and permissions

Use ExtractionConfig for configurable parameters

Return candidates that will be filtered by the noise filtering module

Follow the pattern established in src/extraction/pe_resources.rs and src/extraction/macho_load_commands.rs for consistency.

📄 src/extraction/filters.rs ^(NEW) 🔗

References

📄 .kiro/specs/stringy-binary-analyzer/design.md

📄 src/types.rs ^(MODIFY)

Create the noise filtering module with modular heuristic filters:

Filter Trait (NoiseFilter):

Define trait with method calculate_confidence(text: &str, context: &FilterContext) -> f32

Return confidence score between 0.0 (definitely noise) and 1.0 (definitely legitimate)

Allow filters to be composed and weighted

FilterContext Structure:

Store section type, section name, surrounding bytes (for context)

Include section weight from container parser

Provide access to section permissions (executable, writable)

Character Distribution Filter (CharDistributionFilter):

Calculate character frequency distribution

Detect abnormal ratios: >80% punctuation, >90% same character, >70% non-alphanumeric

Check for reasonable mix of character types

Return low confidence for strings with extreme distributions

Entropy Filter (EntropyFilter):

Use entropy::shannon_entropy to calculate information content

Flag very low entropy (<1.5 bits/byte) as likely padding/repetition

Flag very high entropy (>7.5 bits/byte) as likely binary noise

Optimal range for text: 3.5-6.0 bits/byte

Make thresholds configurable

Linguistic Pattern Filter (LinguisticFilter):

Calculate vowel-to-consonant ratio (reasonable range: 0.2-0.8 for English)

Check for common bigrams (th, he, in, er, an, re, on, at, en, nd)

Detect word-like patterns (alternating consonants and vowels)

Handle non-English strings gracefully (don't penalize too heavily)

Length Filter (LengthFilter):

Penalize excessively long strings (>200 characters) as likely table data

Penalize very short strings in low-weight sections

Adjust thresholds based on section type

Repetition Filter (RepetitionFilter):

Detect repeated character patterns (e.g., "AAAA", "0000", "\x00\x00\x00\x00")

Identify repeated substrings (e.g., "abcabcabc")

Flag padding patterns (spaces, nulls, specific bytes)

Context-Aware Filter (ContextFilter):

Boost confidence for strings in high-weight sections (.rodata, .rdata, __cstring)

Reduce confidence for strings in code sections

Consider section permissions (read-only data more likely to contain strings)

Apply different thresholds based on section type

Composite Filter (CompositeNoiseFilter):

Combine multiple filters with configurable weights

Calculate weighted average of confidence scores

Provide overall confidence assessment

Allow enabling/disabling individual filters

Each filter should be independently testable and configurable. Reference the design document's noise detection requirements (Requirement 5.3) and the NoiseDetector trait pattern from src/classification/ranking.rs (mentioned in design but not yet implemented).

📄 src/extraction/config.rs ^(NEW) 🔗

References

📄 .kiro/specs/stringy-binary-analyzer/design.md

📄 src/types.rs ^(MODIFY)

Create the extraction configuration module:

ExtractionConfig Structure:

min_ascii_length: usize (default: 4) - minimum length for ASCII strings

min_wide_length: usize (default: 3) - minimum length for UTF-16 strings (for future use)

enabled_encodings: Vec<Encoding> - which encodings to extract

noise_filtering_enabled: bool (default: true) - enable/disable noise filtering

min_confidence_threshold: f32 (default: 0.5) - minimum confidence to include string

NoiseFilterConfig Structure:

entropy_min: f32 (default: 1.5) - minimum entropy threshold

entropy_max: f32 (default: 7.5) - maximum entropy threshold

max_length: usize (default: 200) - maximum string length before penalty

max_repetition_ratio: f32 (default: 0.7) - maximum ratio of repeated characters

min_vowel_ratio: f32 (default: 0.1) - minimum vowel ratio for linguistic filter

max_vowel_ratio: f32 (default: 0.9) - maximum vowel ratio for linguistic filter

filter_weights: FilterWeights - weights for combining filter scores

FilterWeights Structure:

entropy_weight: f32 (default: 0.25)

char_distribution_weight: f32 (default: 0.20)

linguistic_weight: f32 (default: 0.20)

length_weight: f32 (default: 0.15)

repetition_weight: f32 (default: 0.10)

context_weight: f32 (default: 0.10)

Ensure weights sum to 1.0

Default Implementation:

Provide sensible defaults based on requirements and design documents

Allow easy customization through builder pattern or direct field access

Validation:

Validate that thresholds are in reasonable ranges

Ensure filter weights sum to 1.0

Provide helpful error messages for invalid configurations

Reference the design document's ExtractionConfig structure and the requirements for configurable thresholds (Requirement 1.4, Acceptance Criteria 4).

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

Update the extraction module to include the new ASCII extraction and filtering modules:

Add module declarations:

pub mod ascii; - ASCII string extraction

pub mod filters; - noise filtering heuristics

pub mod config; - extraction configuration

Re-export key types and functions:

pub use ascii::extract_ascii_strings;

pub use config::{ExtractionConfig, NoiseFilterConfig, FilterWeights};

pub use filters::{NoiseFilter, CompositeNoiseFilter};

Update module documentation:

Document the new ASCII extraction and noise filtering capabilities

Provide usage examples showing how to extract ASCII strings with filtering

Explain the confidence scoring system

Reference the existing PE resource and Mach-O extraction examples

Maintain backward compatibility:

Keep existing exports for extract_resources, extract_resource_strings, and extract_load_command_strings

Ensure the module structure remains consistent with the existing pattern

📄 src/types.rs ^(MODIFY) 🔗

References

📄 src/extraction/pe_resources.rs ^(MODIFY)

📄 src/extraction/macho_load_commands.rs ^(MODIFY)

Extend the types module to support noise filtering:

Add confidence field to FoundString structure:

pub confidence: f32 - confidence score from noise filtering (0.0-1.0)

This represents how confident we are that the string is legitimate vs noise

Default to 1.0 for strings from known-good sources (imports, exports, resources)

Set based on heuristic analysis for section-extracted strings

Update FoundString documentation:

Explain the confidence field and its interpretation

Document that confidence affects final scoring/ranking

Note that confidence is separate from the score field (which is for ranking)

Ensure backward compatibility:

Existing code that creates FoundString instances will need to set confidence

Update all existing FoundString creation sites in src/extraction/pe_resources.rs and src/extraction/macho_load_commands.rs to set confidence: 1.0 (since these are known-good sources)

Consider adding helper methods:

is_high_confidence(&self) -> bool - returns true if confidence >= 0.7

is_low_confidence(&self) -> bool - returns true if confidence < 0.5

Note: The score field is for final ranking (combining section weight, semantic boosts, and noise penalties), while confidence is specifically for the noise filtering assessment.

📄 src/extraction/pe_resources.rs ^(MODIFY) 🔗

References

📄 src/types.rs ^(MODIFY)

Update all FoundString creation sites to include the new confidence field:

In extract_version_info_strings function (around line 438):

Add confidence: 1.0 to the FoundString struct initialization

Version info strings are known-good, so they get maximum confidence

In extract_string_table_strings function (around line 593):

Add confidence: 1.0 to the FoundString struct initialization

String table resources are known-good sources

In manifest extraction (around line 780):

Add confidence: 1.0 to the FoundString struct initialization

Manifest strings are known-good sources

These changes ensure backward compatibility and maintain the semantic meaning that resource strings are high-confidence extractions.

📄 src/extraction/macho_load_commands.rs ^(MODIFY) 🔗

References

📄 src/types.rs ^(MODIFY)

Update all FoundString creation sites to include the new confidence field:

In extract_dylib_strings function (around line 101):

Add confidence: 1.0 to the FoundString struct initialization

Dylib paths from load commands are known-good sources

In extract_rpath_strings function (around line 129):

Add confidence: 1.0 to the FoundString struct initialization

Rpath strings from load commands are known-good sources

These changes maintain consistency with the PE resource extraction pattern and ensure that load command strings are marked as high-confidence.

📄 tests/test_ascii_extraction.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

Create comprehensive unit tests for ASCII extraction:

Basic Extraction Tests:

Test extraction of simple ASCII strings from byte arrays

Verify minimum length threshold is respected

Test null-terminated string handling

Test strings with mixed printable and non-printable characters

Edge Cases:

Empty input data

Data with no valid strings

Strings at section boundaries

Very long strings (>200 characters)

Single-character sequences

Section Context Tests:

Test extraction from different section types (.rodata, .data, .text)

Verify section metadata is properly attached to extracted strings

Test that section weights influence extraction decisions

Configuration Tests:

Test different minimum length thresholds

Test enabling/disabling noise filtering

Test custom confidence thresholds

Follow the testing patterns from tests/integration_pe.rs and use insta for snapshot testing where appropriate.

📄 tests/test_noise_filters.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

Create comprehensive unit tests for noise filtering heuristics:

Character Distribution Filter Tests:

Test detection of all-punctuation strings

Test detection of repeated character strings ("AAAA", "0000")

Test normal text strings (should pass)

Test mixed alphanumeric strings

Entropy Filter Tests:

Test low-entropy strings (padding, repetition): "\x00\x00\x00\x00", "AAAA"

Test high-entropy strings (random binary): random byte sequences

Test normal text entropy: "Hello, World!", "Error: file not found"

Test edge cases at threshold boundaries

Linguistic Filter Tests:

Test English-like strings with good vowel ratios: "Hello world", "Error message"

Test consonant-heavy strings: "bcdfghjklmnpqrstvwxyz"

Test vowel-heavy strings: "aeiouaeiou"

Test non-English strings (should not be overly penalized)

Test strings with numbers and symbols

Length Filter Tests:

Test very short strings (1-3 characters)

Test normal length strings (4-50 characters)

Test long strings (50-200 characters)

Test excessively long strings (>200 characters)

Repetition Filter Tests:

Test repeated characters: "AAAA", "0000", " "

Test repeated patterns: "abcabcabc", "123123123"

Test normal strings with some repetition: "Hello", "Mississippi"

Context Filter Tests:

Test strings from high-weight sections (.rodata) - should boost confidence

Test strings from low-weight sections (.text) - should reduce confidence

Test strings from writable sections - should apply appropriate adjustments

Composite Filter Tests:

Test combining multiple filters with default weights

Test custom filter weights

Test enabling/disabling individual filters

Test overall confidence calculation

Real-World Scenarios:

Test legitimate strings that should pass all filters

Test obvious noise that should fail multiple filters

Test edge cases that might pass some filters but fail others

Test strings from actual binary samples

Use test fixtures with known-good and known-bad strings. Include examples from the issue description (binary noise, padding data, table data, low entropy content).

📄 tests/test_ascii_integration.rs ^(NEW) 🔗

References

📄 tests/integration_pe.rs

📄 tests/integration_elf.rs

Create integration tests for end-to-end ASCII extraction with noise filtering:

Binary Fixture Tests:

Use existing test binaries (test_binary_elf, test_binary_pe.exe, test_binary_macho)

Extract ASCII strings from each binary format

Verify that legitimate strings are extracted with high confidence

Verify that noise is filtered out or marked with low confidence

False Positive Reduction Tests:

Create test data with known noise patterns

Verify that noise is correctly identified and filtered

Measure false positive rate (target: >70% reduction per acceptance criteria)

True Positive Retention Tests:

Create test data with known legitimate strings

Verify that legitimate strings are retained

Measure true positive retention rate (target: >95% per acceptance criteria)

Performance Tests:

Measure extraction time with and without noise filtering

Verify that overhead is <10% (per acceptance criteria)

Test with various binary sizes

Snapshot Tests:

Use insta to create snapshots of extracted strings

Verify consistency across runs

Document expected output for regression testing

Follow the pattern from tests/integration_pe.rs for fixture handling and snapshot testing.

📄 benches/ascii_extraction.rs ^(NEW) 🔗

References

📄 benches/pe.rs

Create performance benchmarks for ASCII extraction and noise filtering:

Basic Extraction Benchmark:

Benchmark ASCII string extraction without filtering

Use test binary fixtures of various sizes

Measure throughput (bytes/second)

Filtered Extraction Benchmark:

Benchmark ASCII string extraction with full noise filtering

Compare against baseline extraction time

Verify <10% overhead target

Individual Filter Benchmarks:

Benchmark each filter independently

Identify performance bottlenecks

Measure filter execution time per string

Composite Filter Benchmark:

Benchmark the combined filter with all heuristics enabled

Test with different filter weight configurations

Measure impact of enabling/disabling individual filters

Entropy Calculation Benchmark:

Benchmark the entropy::shannon_entropy function

Test with strings of various lengths

Verify it's not a performance bottleneck

Large Binary Benchmark:

Test with large binary files (10MB, 50MB, 100MB)

Measure memory usage

Verify scalability

Follow the pattern from benches/pe.rs using criterion for benchmarking. Use black_box to prevent compiler optimizations from skewing results.

📄 docs/src/string-extraction.md ^(MODIFY) 🔗

Update the string extraction documentation to include ASCII noise filtering:

Add ASCII Extraction Section:

Document the ASCII string extraction process

Explain the minimum length threshold

Describe how strings are identified in binary data

Add Noise Filtering Section:

Document the multi-layered heuristic filtering system

Explain each filter type and its purpose:

Character distribution analysis

Entropy-based filtering

Linguistic pattern detection

Length-based filtering

Repetition detection

Context-aware filtering

Provide examples of strings that pass vs fail each filter

Add Configuration Section:

Document the ExtractionConfig and NoiseFilterConfig structures

Explain configurable thresholds and their defaults

Provide examples of customizing filter behavior

Document filter weights and how to adjust them

Add Confidence Scoring Section:

Explain the confidence score (0.0-1.0)

Describe how confidence is calculated from multiple heuristics

Document the relationship between confidence and final ranking

Provide guidance on interpreting confidence scores

Add Performance Section:

Document the performance characteristics of noise filtering

Explain the <10% overhead target

Provide tips for optimizing extraction performance

Add Examples:

Show code examples of using ASCII extraction with filtering

Demonstrate customizing filter configuration

Show how to filter results by confidence threshold

Reference the existing documentation structure and maintain consistency with the style used in other documentation files.

Diagram

sequenceDiagram
    participant User
    participant Extractor as ASCII Extractor
    participant Filters as Noise Filters
    participant Config as Configuration
    participant Section as Section Info
    
    User->>Config: Create ExtractionConfig
    Config-->>User: Config with thresholds
    
    User->>Extractor: extract_ascii_strings(data, section, config)
    Extractor->>Section: Get section type & weight
    Section-->>Extractor: Section metadata
    
    loop For each byte sequence
        Extractor->>Extractor: Scan for printable ASCII
        Extractor->>Extractor: Check minimum length
        alt String meets criteria
            Extractor->>Filters: Apply noise filters
            Filters->>Filters: Character distribution
            Filters->>Filters: Entropy calculation
            Filters->>Filters: Linguistic patterns
            Filters->>Filters: Length check
            Filters->>Filters: Repetition detection
            Filters->>Filters: Context awareness
            Filters-->>Extractor: Confidence score (0.0-1.0)
            
            alt Confidence >= threshold
                Extractor->>Extractor: Create FoundString
                Note over Extractor: Include confidence,<br/>section, offset, tags
            else Confidence < threshold
                Extractor->>Extractor: Discard as noise
            end
        end
    end
    
    Extractor-->>User: Vec<FoundString>

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established architecture with complete container parsing (ELF, PE, Mach-O) and format-specific extraction modules (PE resources, Mach-O load commands). The `FoundString` type is fully defined with fields for text, encoding, offset, RVA, section, length, tags, score, and source. The classification module exists but is empty. No ASCII extraction or noise filtering implementation exists yet. The project uses `goblin` for binary parsing, `pelite` for PE resources, and has comprehensive testing with `insta` snapshots and `criterion` benchmarks.

### Approach

Implement ASCII string noise filtering as a multi-layered system: (1) Create basic ASCII extraction with configurable minimum length, (2) Implement modular heuristic filters for character distribution, entropy, linguistic patterns, and context awareness, (3) Add a confidence scoring system that combines multiple heuristics, (4) Make thresholds configurable through a new configuration struct, (5) Integrate with existing section metadata for context-aware filtering. Use the `entropy` crate for Shannon entropy calculations and follow existing patterns from PE resource and Mach-O extraction modules.

### Reasoning

Explored the project structure, examined existing extraction modules (PE resources, Mach-O load commands), reviewed the types and container parsers, checked the requirements and design documents, and researched available Rust entropy calculation crates.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant Extractor as ASCII Extractor
    participant Filters as Noise Filters
    participant Config as Configuration
    participant Section as Section Info
    
    User->>Config: Create ExtractionConfig
    Config-->>User: Config with thresholds
    
    User->>Extractor: extract_ascii_strings(data, section, config)
    Extractor->>Section: Get section type & weight
    Section-->>Extractor: Section metadata
    
    loop For each byte sequence
        Extractor->>Extractor: Scan for printable ASCII
        Extractor->>Extractor: Check minimum length
        alt String meets criteria
            Extractor->>Filters: Apply noise filters
            Filters->>Filters: Character distribution
            Filters->>Filters: Entropy calculation
            Filters->>Filters: Linguistic patterns
            Filters->>Filters: Length check
            Filters->>Filters: Repetition detection
            Filters->>Filters: Context awareness
            Filters-->>Extractor: Confidence score (0.0-1.0)
            
            alt Confidence >= threshold
                Extractor->>Extractor: Create FoundString
                Note over Extractor: Include confidence,<br/>section, offset, tags
            else Confidence < threshold
                Extractor->>Extractor: Discard as noise
            end
        end
    end
    
    Extractor-->>User: Vec<FoundString>

## Proposed File Changes

### Cargo.toml(MODIFY)

Add the `entropy` crate dependency (version 0.4 or latest) to enable Shannon entropy calculations for noise detection. This lightweight crate provides `shannon_entropy` function that takes byte slices and returns bits per byte, which will be used in the character distribution analysis heuristic.
Add the new benchmark to the Cargo.toml:

1. Add benchmark entry:
   ```toml
   [[bench]]
   name = "ascii_extraction"
   harness = false
   ```

This follows the existing pattern for the `elf` and `pe` benchmarks already defined in the file.

### src/extraction/ascii.rs(NEW)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)
- src/types.rs(MODIFY)

Create the ASCII string extraction module with the following components:

1. **Basic Extraction Function** (`extract_ascii_strings`):
   - Accept parameters: binary data slice, section info, and extraction config
   - Scan for contiguous runs of printable ASCII characters (0x20-0x7E)
   - Apply minimum length threshold from config (default 4 characters)
   - Return vector of candidate strings with their offsets and section context

2. **String Candidate Structure** (`AsciiCandidate`):
   - Store raw text, file offset, section reference, and byte length
   - Include preliminary metadata needed for filtering

3. **Extraction Algorithm**:
   - Iterate through section data byte-by-byte
   - Track current string being built
   - On printable character: append to current string
   - On non-printable: finalize current string if it meets minimum length
   - Handle null-terminated strings properly
   - Respect section boundaries

4. **Integration Points**:
   - Accept `SectionInfo` reference to access section type, weight, and permissions
   - Use `ExtractionConfig` for configurable parameters
   - Return candidates that will be filtered by the noise filtering module

Follow the pattern established in `src/extraction/pe_resources.rs` and `src/extraction/macho_load_commands.rs` for consistency.

### src/extraction/filters.rs(NEW)

References: 

- .kiro/specs/stringy-binary-analyzer/design.md
- src/types.rs(MODIFY)

Create the noise filtering module with modular heuristic filters:

1. **Filter Trait** (`NoiseFilter`):
   - Define trait with method `calculate_confidence(text: &str, context: &FilterContext) -> f32`
   - Return confidence score between 0.0 (definitely noise) and 1.0 (definitely legitimate)
   - Allow filters to be composed and weighted

2. **FilterContext Structure**:
   - Store section type, section name, surrounding bytes (for context)
   - Include section weight from container parser
   - Provide access to section permissions (executable, writable)

3. **Character Distribution Filter** (`CharDistributionFilter`):
   - Calculate character frequency distribution
   - Detect abnormal ratios: >80% punctuation, >90% same character, >70% non-alphanumeric
   - Check for reasonable mix of character types
   - Return low confidence for strings with extreme distributions

4. **Entropy Filter** (`EntropyFilter`):
   - Use `entropy::shannon_entropy` to calculate information content
   - Flag very low entropy (<1.5 bits/byte) as likely padding/repetition
   - Flag very high entropy (>7.5 bits/byte) as likely binary noise
   - Optimal range for text: 3.5-6.0 bits/byte
   - Make thresholds configurable

5. **Linguistic Pattern Filter** (`LinguisticFilter`):
   - Calculate vowel-to-consonant ratio (reasonable range: 0.2-0.8 for English)
   - Check for common bigrams (th, he, in, er, an, re, on, at, en, nd)
   - Detect word-like patterns (alternating consonants and vowels)
   - Handle non-English strings gracefully (don't penalize too heavily)

6. **Length Filter** (`LengthFilter`):
   - Penalize excessively long strings (>200 characters) as likely table data
   - Penalize very short strings in low-weight sections
   - Adjust thresholds based on section type

7. **Repetition Filter** (`RepetitionFilter`):
   - Detect repeated character patterns (e.g., "AAAA", "0000", "\x00\x00\x00\x00")
   - Identify repeated substrings (e.g., "abcabcabc")
   - Flag padding patterns (spaces, nulls, specific bytes)

8. **Context-Aware Filter** (`ContextFilter`):
   - Boost confidence for strings in high-weight sections (.rodata, .rdata, __cstring)
   - Reduce confidence for strings in code sections
   - Consider section permissions (read-only data more likely to contain strings)
   - Apply different thresholds based on section type

9. **Composite Filter** (`CompositeNoiseFilter`):
   - Combine multiple filters with configurable weights
   - Calculate weighted average of confidence scores
   - Provide overall confidence assessment
   - Allow enabling/disabling individual filters

Each filter should be independently testable and configurable. Reference the design document's noise detection requirements (Requirement 5.3) and the NoiseDetector trait pattern from `src/classification/ranking.rs` (mentioned in design but not yet implemented).

### src/extraction/config.rs(NEW)

References: 

- .kiro/specs/stringy-binary-analyzer/design.md
- src/types.rs(MODIFY)

Create the extraction configuration module:

1. **ExtractionConfig Structure**:
   - `min_ascii_length: usize` (default: 4) - minimum length for ASCII strings
   - `min_wide_length: usize` (default: 3) - minimum length for UTF-16 strings (for future use)
   - `enabled_encodings: Vec<Encoding>` - which encodings to extract
   - `noise_filtering_enabled: bool` (default: true) - enable/disable noise filtering
   - `min_confidence_threshold: f32` (default: 0.5) - minimum confidence to include string

2. **NoiseFilterConfig Structure**:
   - `entropy_min: f32` (default: 1.5) - minimum entropy threshold
   - `entropy_max: f32` (default: 7.5) - maximum entropy threshold
   - `max_length: usize` (default: 200) - maximum string length before penalty
   - `max_repetition_ratio: f32` (default: 0.7) - maximum ratio of repeated characters
   - `min_vowel_ratio: f32` (default: 0.1) - minimum vowel ratio for linguistic filter
   - `max_vowel_ratio: f32` (default: 0.9) - maximum vowel ratio for linguistic filter
   - `filter_weights: FilterWeights` - weights for combining filter scores

3. **FilterWeights Structure**:
   - `entropy_weight: f32` (default: 0.25)
   - `char_distribution_weight: f32` (default: 0.20)
   - `linguistic_weight: f32` (default: 0.20)
   - `length_weight: f32` (default: 0.15)
   - `repetition_weight: f32` (default: 0.10)
   - `context_weight: f32` (default: 0.10)
   - Ensure weights sum to 1.0

4. **Default Implementation**:
   - Provide sensible defaults based on requirements and design documents
   - Allow easy customization through builder pattern or direct field access

5. **Validation**:
   - Validate that thresholds are in reasonable ranges
   - Ensure filter weights sum to 1.0
   - Provide helpful error messages for invalid configurations

Reference the design document's ExtractionConfig structure and the requirements for configurable thresholds (Requirement 1.4, Acceptance Criteria 4).

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)

Update the extraction module to include the new ASCII extraction and filtering modules:

1. Add module declarations:
   - `pub mod ascii;` - ASCII string extraction
   - `pub mod filters;` - noise filtering heuristics
   - `pub mod config;` - extraction configuration

2. Re-export key types and functions:
   - `pub use ascii::extract_ascii_strings;`
   - `pub use config::{ExtractionConfig, NoiseFilterConfig, FilterWeights};`
   - `pub use filters::{NoiseFilter, CompositeNoiseFilter};`

3. Update module documentation:
   - Document the new ASCII extraction and noise filtering capabilities
   - Provide usage examples showing how to extract ASCII strings with filtering
   - Explain the confidence scoring system
   - Reference the existing PE resource and Mach-O extraction examples

4. Maintain backward compatibility:
   - Keep existing exports for `extract_resources`, `extract_resource_strings`, and `extract_load_command_strings`
   - Ensure the module structure remains consistent with the existing pattern

### src/types.rs(MODIFY)

References: 

- src/extraction/pe_resources.rs(MODIFY)
- src/extraction/macho_load_commands.rs(MODIFY)

Extend the types module to support noise filtering:

1. Add `confidence` field to `FoundString` structure:
   - `pub confidence: f32` - confidence score from noise filtering (0.0-1.0)
   - This represents how confident we are that the string is legitimate vs noise
   - Default to 1.0 for strings from known-good sources (imports, exports, resources)
   - Set based on heuristic analysis for section-extracted strings

2. Update `FoundString` documentation:
   - Explain the confidence field and its interpretation
   - Document that confidence affects final scoring/ranking
   - Note that confidence is separate from the score field (which is for ranking)

3. Ensure backward compatibility:
   - Existing code that creates `FoundString` instances will need to set confidence
   - Update all existing `FoundString` creation sites in `src/extraction/pe_resources.rs` and `src/extraction/macho_load_commands.rs` to set `confidence: 1.0` (since these are known-good sources)

4. Consider adding helper methods:
   - `is_high_confidence(&self) -> bool` - returns true if confidence >= 0.7
   - `is_low_confidence(&self) -> bool` - returns true if confidence < 0.5

Note: The `score` field is for final ranking (combining section weight, semantic boosts, and noise penalties), while `confidence` is specifically for the noise filtering assessment.

### src/extraction/pe_resources.rs(MODIFY)

References: 

- src/types.rs(MODIFY)

Update all `FoundString` creation sites to include the new `confidence` field:

1. In `extract_version_info_strings` function (around line 438):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Version info strings are known-good, so they get maximum confidence

2. In `extract_string_table_strings` function (around line 593):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - String table resources are known-good sources

3. In manifest extraction (around line 780):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Manifest strings are known-good sources

These changes ensure backward compatibility and maintain the semantic meaning that resource strings are high-confidence extractions.

### src/extraction/macho_load_commands.rs(MODIFY)

References: 

- src/types.rs(MODIFY)

Update all `FoundString` creation sites to include the new `confidence` field:

1. In `extract_dylib_strings` function (around line 101):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Dylib paths from load commands are known-good sources

2. In `extract_rpath_strings` function (around line 129):
   - Add `confidence: 1.0` to the `FoundString` struct initialization
   - Rpath strings from load commands are known-good sources

These changes maintain consistency with the PE resource extraction pattern and ensure that load command strings are marked as high-confidence.

### tests/test_ascii_extraction.rs(NEW)

References: 

- tests/integration_pe.rs

Create comprehensive unit tests for ASCII extraction:

1. **Basic Extraction Tests**:
   - Test extraction of simple ASCII strings from byte arrays
   - Verify minimum length threshold is respected
   - Test null-terminated string handling
   - Test strings with mixed printable and non-printable characters

2. **Edge Cases**:
   - Empty input data
   - Data with no valid strings
   - Strings at section boundaries
   - Very long strings (>200 characters)
   - Single-character sequences

3. **Section Context Tests**:
   - Test extraction from different section types (.rodata, .data, .text)
   - Verify section metadata is properly attached to extracted strings
   - Test that section weights influence extraction decisions

4. **Configuration Tests**:
   - Test different minimum length thresholds
   - Test enabling/disabling noise filtering
   - Test custom confidence thresholds

Follow the testing patterns from `tests/integration_pe.rs` and use `insta` for snapshot testing where appropriate.

### tests/test_noise_filters.rs(NEW)

References: 

- tests/integration_pe.rs

Create comprehensive unit tests for noise filtering heuristics:

1. **Character Distribution Filter Tests**:
   - Test detection of all-punctuation strings
   - Test detection of repeated character strings ("AAAA", "0000")
   - Test normal text strings (should pass)
   - Test mixed alphanumeric strings

2. **Entropy Filter Tests**:
   - Test low-entropy strings (padding, repetition): "\x00\x00\x00\x00", "AAAA"
   - Test high-entropy strings (random binary): random byte sequences
   - Test normal text entropy: "Hello, World!", "Error: file not found"
   - Test edge cases at threshold boundaries

3. **Linguistic Filter Tests**:
   - Test English-like strings with good vowel ratios: "Hello world", "Error message"
   - Test consonant-heavy strings: "bcdfghjklmnpqrstvwxyz"
   - Test vowel-heavy strings: "aeiouaeiou"
   - Test non-English strings (should not be overly penalized)
   - Test strings with numbers and symbols

4. **Length Filter Tests**:
   - Test very short strings (1-3 characters)
   - Test normal length strings (4-50 characters)
   - Test long strings (50-200 characters)
   - Test excessively long strings (>200 characters)

5. **Repetition Filter Tests**:
   - Test repeated characters: "AAAA", "0000", "    "
   - Test repeated patterns: "abcabcabc", "123123123"
   - Test normal strings with some repetition: "Hello", "Mississippi"

6. **Context Filter Tests**:
   - Test strings from high-weight sections (.rodata) - should boost confidence
   - Test strings from low-weight sections (.text) - should reduce confidence
   - Test strings from writable sections - should apply appropriate adjustments

7. **Composite Filter Tests**:
   - Test combining multiple filters with default weights
   - Test custom filter weights
   - Test enabling/disabling individual filters
   - Test overall confidence calculation

8. **Real-World Scenarios**:
   - Test legitimate strings that should pass all filters
   - Test obvious noise that should fail multiple filters
   - Test edge cases that might pass some filters but fail others
   - Test strings from actual binary samples

Use test fixtures with known-good and known-bad strings. Include examples from the issue description (binary noise, padding data, table data, low entropy content).

### tests/test_ascii_integration.rs(NEW)

References: 

- tests/integration_pe.rs
- tests/integration_elf.rs

Create integration tests for end-to-end ASCII extraction with noise filtering:

1. **Binary Fixture Tests**:
   - Use existing test binaries (test_binary_elf, test_binary_pe.exe, test_binary_macho)
   - Extract ASCII strings from each binary format
   - Verify that legitimate strings are extracted with high confidence
   - Verify that noise is filtered out or marked with low confidence

2. **False Positive Reduction Tests**:
   - Create test data with known noise patterns
   - Verify that noise is correctly identified and filtered
   - Measure false positive rate (target: >70% reduction per acceptance criteria)

3. **True Positive Retention Tests**:
   - Create test data with known legitimate strings
   - Verify that legitimate strings are retained
   - Measure true positive retention rate (target: >95% per acceptance criteria)

4. **Performance Tests**:
   - Measure extraction time with and without noise filtering
   - Verify that overhead is <10% (per acceptance criteria)
   - Test with various binary sizes

5. **Snapshot Tests**:
   - Use `insta` to create snapshots of extracted strings
   - Verify consistency across runs
   - Document expected output for regression testing

Follow the pattern from `tests/integration_pe.rs` for fixture handling and snapshot testing.

### benches/ascii_extraction.rs(NEW)

References: 

- benches/pe.rs

Create performance benchmarks for ASCII extraction and noise filtering:

1. **Basic Extraction Benchmark**:
   - Benchmark ASCII string extraction without filtering
   - Use test binary fixtures of various sizes
   - Measure throughput (bytes/second)

2. **Filtered Extraction Benchmark**:
   - Benchmark ASCII string extraction with full noise filtering
   - Compare against baseline extraction time
   - Verify <10% overhead target

3. **Individual Filter Benchmarks**:
   - Benchmark each filter independently
   - Identify performance bottlenecks
   - Measure filter execution time per string

4. **Composite Filter Benchmark**:
   - Benchmark the combined filter with all heuristics enabled
   - Test with different filter weight configurations
   - Measure impact of enabling/disabling individual filters

5. **Entropy Calculation Benchmark**:
   - Benchmark the `entropy::shannon_entropy` function
   - Test with strings of various lengths
   - Verify it's not a performance bottleneck

6. **Large Binary Benchmark**:
   - Test with large binary files (10MB, 50MB, 100MB)
   - Measure memory usage
   - Verify scalability

Follow the pattern from `benches/pe.rs` using `criterion` for benchmarking. Use `black_box` to prevent compiler optimizations from skewing results.

### docs/src/string-extraction.md(MODIFY)

Update the string extraction documentation to include ASCII noise filtering:

1. **Add ASCII Extraction Section**:
   - Document the ASCII string extraction process
   - Explain the minimum length threshold
   - Describe how strings are identified in binary data

2. **Add Noise Filtering Section**:
   - Document the multi-layered heuristic filtering system
   - Explain each filter type and its purpose:
     - Character distribution analysis
     - Entropy-based filtering
     - Linguistic pattern detection
     - Length-based filtering
     - Repetition detection
     - Context-aware filtering
   - Provide examples of strings that pass vs fail each filter

3. **Add Configuration Section**:
   - Document the `ExtractionConfig` and `NoiseFilterConfig` structures
   - Explain configurable thresholds and their defaults
   - Provide examples of customizing filter behavior
   - Document filter weights and how to adjust them

4. **Add Confidence Scoring Section**:
   - Explain the confidence score (0.0-1.0)
   - Describe how confidence is calculated from multiple heuristics
   - Document the relationship between confidence and final ranking
   - Provide guidance on interpreting confidence scores

5. **Add Performance Section**:
   - Document the performance characteristics of noise filtering
   - Explain the <10% overhead target
   - Provide tips for optimizing extraction performance

6. **Add Examples**:
   - Show code examples of using ASCII extraction with filtering
   - Demonstrate customizing filter configuration
   - Show how to filter results by confidence threshold

Reference the existing documentation structure and maintain consistency with the style used in other documentation files.

Developer Humor

🎯 The String Detective's Creed\n>\n> In a world of binary noise and padding bytes,\n> Where garbage strings give analysts the frights,\n> We filter with entropy, linguistics, and care,\n> To find the real strings hiding in there!\n>\n> No more \AAAA\ or random hex debris,\n> Just meaningful text, as clean as can be! 🔍✨

Execution Information

Branch: main
Commit: baf4526

💡 Tips

Supported Commands (Inside Comments)

Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Implement ASCII String Noise Filtering with Intelligent Heuristics #10

Description

Summary

Problem Context

Proposed Solution

1. Character Distribution Analysis

2. Linguistic Patterns

3. Context-Aware Filtering

4. Configurable Thresholds

5. Scoring System

Acceptance Criteria

Technical Notes

Requirements

Dependencies

Task-ID

Activity

unclesp1d3r commented on Sep 30, 2025

coderabbitai commented on Sep 30, 2025

Changes Made:

📝 Title

📄 Body Enhancements

🏷️ Labels Added

unclesp1d3r commented on Nov 10, 2025

coderabbitai commented on Nov 10, 2025

✅ Current Status

📝 Title

📄 Body Sections

🏷️ Labels Applied

👤 Assigned To

unclesp1d3r commented on Nov 12, 2025

traycerai commented on Nov 12, 2025

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

traycerai commented on Nov 12, 2025

Plan

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions