Implement Noise Penalty Detection for String Quality Scoring

Overview

Implement a noise penalty detection system that identifies and penalizes low-quality strings during the ranking phase. This feature is critical for ensuring that StringyMcStringFace surfaces meaningful strings while filtering out garbage data, padding bytes, encrypted content, and table artifacts that plague traditional strings output.

Problem Statement

Binary files contain various types of noise that should be deprioritized or filtered:

High Entropy Strings: Random-looking data from encryption, compression, or binary blobs (e.g., xK9#mP@vQ2%nR)
Excessive Length: Extremely long strings are often padding, base64 blobs, or table data rather than meaningful text
Repeated Patterns: Padding sequences (AAAA..., \x00\x00...) or table delimiters that add no semantic value
Table Data: Structured binary data that appears as strings but contains mostly non-printable or low-information content

Without noise detection, the ranking system will surface these low-value strings alongside genuinely useful data like URLs, file paths, and error messages.

Proposed Solution

High Entropy Detection

Calculate Shannon entropy for each extracted string and apply penalties for entropy above threshold:

fn calculate_entropy(s: &str) -> f64 {
    let mut frequencies = HashMap::new();
    for c in s.chars() {
        *frequencies.entry(c).or_insert(0) += 1;
    }
    
    let len = s.len() as f64;
    frequencies.values()
        .map(|&count| {
            let p = count as f64 / len;
            -p * p.log2()
        })
        .sum()
}

// Apply penalty if entropy > 4.5 (tunable threshold)
// Natural language typically has entropy 3.5-4.5
// Random data approaches 8.0 for printable ASCII

Penalty Approach:

Entropy < 4.0: No penalty (natural language)
Entropy 4.0-5.0: Linear penalty 0-30 points
Entropy > 5.0: Fixed penalty -30 points

Excessive Length Penalty

Penalize strings beyond reasonable lengths using a tiered approach:

fn length_penalty(len: usize) -> i32 {
    match len {
        0..=256 => 0,           // Normal strings
        257..=512 => -10,       // Slightly long
        513..=1024 => -20,      // Very long (likely base64/table)
        _ => -30                // Extremely long (definitely noise)
    }
}

Repeated Pattern Detection

Detect strings dominated by repeated characters or short sequences:

fn detect_repeated_patterns(s: &str) -> Option<i32> {
    // Check for single character repetition
    let unique_chars = s.chars().collect::<HashSet<_>>().len();
    let repetition_ratio = unique_chars as f64 / s.len() as f64;
    
    if repetition_ratio < 0.1 {
        return Some(-25); // >90% same character
    }
    
    // Check for short repeating sequences (2-4 chars)
    // e.g., "ABABAB", "123123123"
    for pattern_len in 2..=4 {
        if is_repeating_pattern(s, pattern_len) {
            return Some(-20);
        }
    }
    
    None
}

Table Data Heuristics

Detect table-like data using multiple indicators:

Delimiter Density: High frequency of delimiters (commas, pipes, tabs, nulls)
Binary Interleaving: Printable characters interspersed with non-printable
Low Alphanumeric Ratio: Less than 60% alphanumeric content

fn is_table_data(s: &str) -> bool {
    let delimiter_count = s.chars().filter(|&c| matches!(c, ',' | '|' | '\t' | '\0')).count();
    let delimiter_ratio = delimiter_count as f64 / s.len() as f64;
    
    let alphanum_count = s.chars().filter(|c| c.is_alphanumeric()).count();
    let alphanum_ratio = alphanum_count as f64 / s.len() as f64;
    
    delimiter_ratio > 0.3 || alphanum_ratio < 0.6
}

Implementation Plan

Add noise detection module (src/ranker/noise.rs)
- Entropy calculation function
- Length penalty function
- Pattern detection functions
- Table data heuristics
Integrate with ranking system
- Add noise penalty field to ExtractedString score calculation
- Apply penalties during ranking phase
- Ensure penalties are composable (multiple penalties can apply)
Configuration
- Make thresholds tunable via CLI flags or config file
- Default conservative values that work for most binaries
Testing
- Unit tests for each detection function with known inputs
- Integration tests with real binary samples
- Snapshot tests to verify penalty application

Acceptance Criteria

Entropy calculation implemented with configurable threshold
Length penalty with tiered approach (256/512/1024 byte thresholds)
Repeated pattern detection for single chars and short sequences
Table data heuristics with delimiter and alphanumeric ratio checks
Integration with ranking system (penalties affect final scores)
Unit tests achieving >90% coverage for noise detection module
Documentation with examples of penalized vs. non-penalized strings
Performance: Noise detection adds <10% overhead to overall extraction time

Technical Considerations

Performance: Entropy calculation is O(n) per string; consider caching for reused strings
Encoding-Aware: UTF-16 strings need byte-pair entropy, not character entropy
Threshold Tuning: Default values should be validated against diverse binary corpus (ELF, PE, Mach-O)
False Positives: Some legitimate strings may have high entropy (e.g., crypto keys, hashes); use context tags to avoid over-penalizing

Dependencies

Blocked by: Ranking System Foundation (task: stringy-analyzer/ranking-system)
Requires: ExtractedString type with score field
Enables: High-quality output that rivals manual analysis

Requirements

Requirement: 5.3

Task ID

stringy-analyzer/noise-penalty-detection

#!/bin/bash
# Fetch the current issue details
gh issue view 24 --json title,body,labels,assignees,state,milestone

# Get a better understanding of the repository structure
fd -t f README.md -x cat {}

# Look for existing code related to ranking or penalty systems
fd -e py -e rb -e js -e ts | head -20

Length of output: 9830

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced content
gh issue edit 24 \
  --title "Implement Noise Penalty Detection for String Quality Scoring" \
  --body "## Overview

Implement a noise penalty detection system that identifies and penalizes low-quality strings during the ranking phase. This feature is critical for ensuring that StringyMcStringFace surfaces meaningful strings while filtering out garbage data, padding bytes, encrypted content, and table artifacts that plague traditional \`strings\` output.

## Problem Statement

Binary files contain various types of noise that should be deprioritized or filtered:

1. **High Entropy Strings**: Random-looking data from encryption, compression, or binary blobs (e.g., \`xK9#mP@vQ2$wL8%nR\`)
2. **Excessive Length**: Extremely long strings are often padding, base64 blobs, or table data rather than meaningful text
3. **Repeated Patterns**: Padding sequences (\`AAAA...\`, \`\\x00\\x00...\`) or table delimiters that add no semantic value
4. **Table Data**: Structured binary data that appears as strings but contains mostly non-printable or low-information content

Without noise detection, the ranking system will surface these low-value strings alongside genuinely useful data like URLs, file paths, and error messages.

## Proposed Solution

### High Entropy Detection

Calculate Shannon entropy for each extracted string and apply penalties for entropy above threshold:

\`\`\`rust
fn calculate_entropy(s: &str) -> f64 {
    let mut frequencies = HashMap::new();
    for c in s.chars() {
        *frequencies.entry(c).or_insert(0) += 1;
    }
    
    let len = s.len() as f64;
    frequencies.values()
        .map(|&count| {
            let p = count as f64 / len;
            -p * p.log2()
        })
        .sum()
}

// Apply penalty if entropy > 4.5 (tunable threshold)
// Natural language typically has entropy 3.5-4.5
// Random data approaches 8.0 for printable ASCII
\`\`\`

**Penalty Approach**:
- Entropy < 4.0: No penalty (natural language)
- Entropy 4.0-5.0: Linear penalty 0-30 points
- Entropy > 5.0: Fixed penalty -30 points

### Excessive Length Penalty

Penalize strings beyond reasonable lengths using a tiered approach:

\`\`\`rust
fn length_penalty(len: usize) -> i32 {
    match len {
        0..=256 => 0,           // Normal strings
        257..=512 => -10,       // Slightly long
        513..=1024 => -20,      // Very long (likely base64/table)
        _ => -30                // Extremely long (definitely noise)
    }
}
\`\`\`

### Repeated Pattern Detection

Detect strings dominated by repeated characters or short sequences:

\`\`\`rust
fn detect_repeated_patterns(s: &str) -> Option<i32> {
    // Check for single character repetition
    let unique_chars = s.chars().collect::<HashSet<_>>().len();
    let repetition_ratio = unique_chars as f64 / s.len() as f64;
    
    if repetition_ratio < 0.1 {
        return Some(-25); // >90% same character
    }
    
    // Check for short repeating sequences (2-4 chars)
    // e.g., \"ABABAB\", \"123123123\"
    for pattern_len in 2..=4 {
        if is_repeating_pattern(s, pattern_len) {
            return Some(-20);
        }
    }
    
    None
}
\`\`\`

### Table Data Heuristics

Detect table-like data using multiple indicators:

- **Delimiter Density**: High frequency of delimiters (commas, pipes, tabs, nulls)
- **Binary Interleaving**: Printable characters interspersed with non-printable
- **Low Alphanumeric Ratio**: Less than 60% alphanumeric content

\`\`\`rust
fn is_table_data(s: &str) -> bool {
    let delimiter_count = s.chars().filter(|&c| matches!(c, ',' | '|' | '\\t' | '\\0')).count();
    let delimiter_ratio = delimiter_count as f64 / s.len() as f64;
    
    let alphanum_count = s.chars().filter(|c| c.is_alphanumeric()).count();
    let alphanum_ratio = alphanum_count as f64 / s.len() as f64;
    
    delimiter_ratio > 0.3 || alphanum_ratio < 0.6
}
\`\`\`

## Implementation Plan

1. **Add noise detection module** (\`src/ranker/noise.rs\`)
   - Entropy calculation function
   - Length penalty function
   - Pattern detection functions
   - Table data heuristics

2. **Integrate with ranking system**
   - Add noise penalty field to \`ExtractedString\` score calculation
   - Apply penalties during ranking phase
   - Ensure penalties are composable (multiple penalties can apply)

3. **Configuration**
   - Make thresholds tunable via CLI flags or config file
   - Default conservative values that work for most binaries

4. **Testing**
   - Unit tests for each detection function with known inputs
   - Integration tests with real binary samples
   - Snapshot tests to verify penalty application

## Acceptance Criteria

- [ ] Entropy calculation implemented with configurable threshold
- [ ] Length penalty with tiered approach (256/512/1024 byte thresholds)
- [ ] Repeated pattern detection for single chars and short sequences
- [ ] Table data heuristics with delimiter and alphanumeric ratio checks
- [ ] Integration with ranking system (penalties affect final scores)
- [ ] Unit tests achieving >90% coverage for noise detection module
- [ ] Documentation with examples of penalized vs. non-penalized strings
- [ ] Performance: Noise detection adds <10% overhead to overall extraction time

## Technical Considerations

- **Performance**: Entropy calculation is O(n) per string; consider caching for reused strings
- **Encoding-Aware**: UTF-16 strings need byte-pair entropy, not character entropy
- **Threshold Tuning**: Default values should be validated against diverse binary corpus (ELF, PE, Mach-O)
- **False Positives**: Some legitimate strings may have high entropy (e.g., crypto keys, hashes); use context tags to avoid over-penalizing

## Dependencies

- **Blocked by**: Ranking System Foundation (task: stringy-analyzer/ranking-system)
- **Requires**: \`ExtractedString\` type with score field
- **Enables**: High-quality output that rivals manual analysis

## Requirements

Requirement: 5.3

## Task ID

\`stringy-analyzer/noise-penalty-detection\`

## Related

This feature directly addresses the \"Eliminates noise\" advantage mentioned in the README, helping StringyMcStringFace stop \"dumping padding, tables, and interleaved garbage\" that plagues standard \`strings\` output."

echo "Issue #24 updated successfully"

Length of output: 92

@unclesp1d3r I've successfully enhanced issue #24! Here's what was updated:

✅ Enhanced Issue Title

Changed from "Noise Penalty Detection" to "Implement Noise Penalty Detection for String Quality Scoring" - making it more descriptive and actionable.

✅ Comprehensive Issue Body

Added extensive context including:

Overview & Problem Statement: Why noise detection is critical for StringyMcStringFace's value proposition
Proposed Technical Solutions with Rust code examples:
- Shannon entropy calculation with configurable thresholds (natural language: 3.5-4.5, random data: ~8.0)
- Tiered length penalties (256/512/1024 byte thresholds)
- Repeated pattern detection (single chars and short sequences)
- Table data heuristics (delimiter density, alphanumeric ratio)
Implementation Plan: 4-step roadmap from module creation to testing
Acceptance Criteria: 8 concrete, measurable checkboxes
Technical Considerations: Performance, encoding-awareness, threshold tuning, false positive mitigation
Dependencies & Requirements: Links to blocking tasks and requirement 5.3

✅ Metadata

The existing labels were already appropriate and have been preserved:

area:analyzer - Correct scope
type:enhancement - Correct type
status:backlog - Current status
lang:rust - Implementation language
needs:tests - Requirement noted

The issue is already assigned to you and in the v0.1 milestone.

View the enhanced issue: #24

The issue now provides a complete technical specification that any contributor can use to implement this feature consistently with the project's goals of eliminating noise from binary string extraction.

Plan

Observations

The codebase has a solid foundation with complete container parsers for ELF, PE, and Mach-O formats. The FoundString type already includes a score: i32 field, but the ranking system doesn't exist yet. Section weights are already implemented in each parser (ranging from 1.0 to 10.0), providing a foundation for the scoring system. The classification and extraction modules are empty placeholders, and there's no ranking module yet. The project uses standard Rust testing with insta for snapshots and criterion for benchmarks.

Approach

Implement a modular noise penalty detection system as part of the ranking infrastructure. Create the ranking module with a NoiseDetector trait for extensibility, then implement specific noise detectors (entropy, length, patterns, table data) as separate components. Integrate these into a RankingEngine that calculates final scores by combining section weights, semantic boosts, and noise penalties. Make thresholds configurable through a RankingConfig struct. Follow the existing codebase patterns: trait-based design, comprehensive unit tests, and clear separation of concerns.

Reasoning

I explored the project structure and found that the core infrastructure (types, container parsers, section weighting) is complete. I examined the design document and requirements to understand the scoring formula: Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty. I reviewed the existing test patterns and confirmed that the ranking module needs to be created from scratch, with noise detection as one of its key components.

Proposed File Changes

📄 src/classification/mod.rs ^(MODIFY) 🔗

References

📄 src/container/mod.rs

Add module declarations for the new ranking and noise detection modules. Export the RankingEngine, RankingConfig, and NoiseDetector trait for use by other parts of the codebase. This file currently only contains a comment, so add:

pub mod ranking; - declares the ranking module

pub mod noise; - declares the noise detection module

Re-export key types: pub use ranking::{RankingEngine, RankingConfig}; and pub use noise::NoiseDetector;

This follows the pattern established in src/container/mod.rs which declares submodules and re-exports parsers.

📄 src/classification/ranking.rs ^(NEW) 🔗

References

📄 src/types.rs

📄 .kiro/specs/stringy-binary-analyzer/design.md

Create the ranking system foundation with the following components:

RankingConfig struct: Configuration for ranking behavior with fields:

section_weight_multiplier: f32 (default 1.0) - scales section weights from container parsers

tag_boosts: HashMap<Tag, i32> - semantic boost values per tag type (URL: +5, GUID: +5, registry: +5, filepath: +3, format string: +3, import/export: +2)

enable_noise_penalties: bool (default true) - toggle noise detection

noise_config: NoiseConfig - configuration for noise detectors

NoiseConfig struct: Nested configuration for noise detection thresholds:

entropy_threshold_low: f64 (default 4.0) - start of entropy penalty range

entropy_threshold_high: f64 (default 5.0) - maximum entropy before fixed penalty

entropy_penalty_max: i32 (default -30) - maximum entropy penalty

length_tier_1: usize (default 256) - first length threshold

length_tier_2: usize (default 512) - second length threshold

length_tier_3: usize (default 1024) - third length threshold

length_penalty_tier_1: i32 (default -10)

length_penalty_tier_2: i32 (default -20)

length_penalty_tier_3: i32 (default -30)

pattern_repetition_threshold: f64 (default 0.1) - ratio below which string is considered repetitive

pattern_penalty: i32 (default -25)

table_delimiter_threshold: f64 (default 0.3) - delimiter ratio for table detection

table_alphanum_threshold: f64 (default 0.6) - alphanumeric ratio for table detection

table_penalty: i32 (default -20)

RankingEngine struct: Main scoring engine with fields:

config: RankingConfig

noise_detectors: Vec<Box<dyn NoiseDetector>> - pluggable noise detectors

RankingEngine implementation:

new(config: RankingConfig) -> Self - constructor that initializes noise detectors based on config

calculate_score(&self, found_string: &FoundString, section_weight: f32) -> i32 - main scoring function that:

Converts section weight to integer score (multiply by 10 to preserve precision)

Adds semantic boosts based on tags

Subtracts noise penalties from all enabled detectors

Returns final score

rank_strings(&self, strings: &mut [FoundString], section_weights: &HashMap<String, f32>) - sorts strings by score in descending order after calculating scores

Default implementations: Provide sensible defaults for RankingConfig and NoiseConfig using Default trait, matching the values specified in the issue.

Include comprehensive unit tests:

Test score calculation with various combinations of section weights, tags, and noise penalties

Test that noise penalties are correctly applied

Test that configuration changes affect scoring

Test ranking order with sample strings

📄 src/classification/noise.rs ^(NEW) 🔗

References

📄 src/types.rs

Create the noise detection module with the NoiseDetector trait and concrete implementations:

NoiseDetector trait: Define the interface for noise detection:

fn calculate_penalty(&self, string: &FoundString) -> i32 - returns penalty value (negative integer)

fn name(&self) -> &str - returns detector name for debugging

EntropyDetector struct: Detects high-entropy (random-looking) strings:

Fields: threshold_low: f64, threshold_high: f64, max_penalty: i32

new(threshold_low: f64, threshold_high: f64, max_penalty: i32) -> Self

calculate_entropy(text: &str) -> f64 - private helper that:

Counts character frequencies in the string

Calculates Shannon entropy: sum of -p * log2(p) for each character probability

Returns entropy value (0.0 to ~8.0 for printable ASCII)

Implementation of NoiseDetector::calculate_penalty:

If entropy < threshold_low: return 0 (no penalty)

If entropy between thresholds: linear interpolation from 0 to max_penalty

If entropy > threshold_high: return max_penalty

LengthDetector struct: Penalizes excessively long strings:

Fields: tier_1: usize, tier_2: usize, tier_3: usize, penalty_1: i32, penalty_2: i32, penalty_3: i32

new(tier_1: usize, tier_2: usize, tier_3: usize, penalty_1: i32, penalty_2: i32, penalty_3: i32) -> Self

Implementation of NoiseDetector::calculate_penalty:

Match on string length and return appropriate penalty based on tier

0..=tier_1: 0 penalty

tier_1+1..=tier_2: penalty_1

tier_2+1..=tier_3: penalty_2

tier_3+1..: penalty_3

PatternDetector struct: Detects repeated character patterns:

Fields: repetition_threshold: f64, penalty: i32

new(repetition_threshold: f64, penalty: i32) -> Self

detect_single_char_repetition(text: &str) -> f64 - calculates ratio of unique chars to total length

is_repeating_pattern(text: &str, pattern_len: usize) -> bool - checks if string is composed of repeating N-character pattern

Implementation of NoiseDetector::calculate_penalty:

Check single character repetition ratio

If ratio < threshold: return penalty

Check for 2-4 character repeating patterns

If found: return penalty

Otherwise: return 0

TableDataDetector struct: Detects table-like structured data:

Fields: delimiter_threshold: f64, alphanum_threshold: f64, penalty: i32

new(delimiter_threshold: f64, alphanum_threshold: f64, penalty: i32) -> Self

calculate_delimiter_ratio(text: &str) -> f64 - counts delimiters (comma, pipe, tab, null) divided by length

calculate_alphanum_ratio(text: &str) -> f64 - counts alphanumeric chars divided by length

Implementation of NoiseDetector::calculate_penalty:

Calculate both ratios

If delimiter_ratio > threshold OR alphanum_ratio < threshold: return penalty

Otherwise: return 0

Helper function: create_default_detectors(config: &NoiseConfig) -> Vec<Box<dyn NoiseDetector>> - factory function that creates all four detector instances with config values.

Include comprehensive unit tests for each detector:

EntropyDetector tests: Test with low-entropy strings ("hello world"), medium-entropy strings ("H3ll0_W0r1d!"), high-entropy strings ("xK9#mP@vQ2%nR")

LengthDetector tests: Test strings at each tier boundary (255, 256, 257, 511, 512, 513, 1023, 1024, 1025 chars)

PatternDetector tests: Test single-char repetition ("AAAAAAA"), 2-char patterns ("ABABAB"), 3-char patterns ("123123123"), normal strings

TableDataDetector tests: Test high-delimiter strings ("a,b,c,d,e,f"), low-alphanum strings ("!!!@@@###$$$"), normal strings

Test that penalties are negative integers

Test edge cases (empty strings, single characters, unicode)

📄 src/lib.rs ^(MODIFY) 🔗

Update the re-exports section to include the new ranking types. After the existing re-export block (around line 51-54), add:
pub use classification::{RankingEngine, RankingConfig, NoiseDetector};
This makes the ranking system accessible to users of the library without requiring them to import from nested modules. Update the module documentation comment (around line 40) to reflect that the classification module now includes ranking functionality:

Change:

classification: Semantic analysis and tagging (🚧 Types defined)

To:

classification: Semantic analysis, tagging, and ranking (🚧 Ranking system implemented)

This follows the existing pattern where commonly used types from submodules are re-exported at the library root level, as seen with the types module exports.

📄 Cargo.toml ^(MODIFY) 🔗

No changes needed. The existing dependencies are sufficient for the noise penalty detection implementation:

Standard library provides HashMap for configuration and frequency counting

No external crates needed for entropy calculation (using standard math)

No external crates needed for pattern detection (using string iteration)

The implementation uses only standard Rust library features. If future enhancements require regex for more sophisticated pattern detection, the regex crate can be added later, but it's not needed for the current specification.

📄 tests/integration_ranking.rs ^(NEW) 🔗

References

📄 tests/integration_elf.rs

📄 src/types.rs

Create integration tests for the ranking system with noise penalty detection:

Test fixtures: Create sample FoundString instances representing different scenarios:

High-quality strings: URLs, file paths, error messages with low entropy and reasonable length

Noisy strings: high-entropy data, extremely long strings, repeated patterns, table data

Mixed strings: legitimate strings with some noise characteristics

Test cases:

test_entropy_penalty_integration: Create strings with varying entropy levels and verify:

Low-entropy strings ("Hello World") receive no penalty

Medium-entropy strings receive proportional penalties

High-entropy strings ("xK9#mP@vQ2%nR") receive maximum penalty

Verify final scores reflect entropy differences

test_length_penalty_integration: Create strings at different length tiers:

Short strings (< 256 chars) receive no length penalty

Medium strings (256-512 chars) receive tier 1 penalty

Long strings (512-1024 chars) receive tier 2 penalty

Very long strings (> 1024 chars) receive tier 3 penalty

Verify scores decrease with length

test_pattern_penalty_integration: Test pattern detection:

Strings with >90% same character ("AAAAAAAAAA") receive pattern penalty

Strings with repeating 2-4 char patterns ("ABABABABAB") receive penalty

Normal strings with varied characters receive no penalty

test_table_data_penalty_integration: Test table detection:

Strings with high delimiter density ("a,b,c,d,e,f,g,h") receive penalty

Strings with low alphanumeric ratio ("!!!@@@###$$$") receive penalty

Normal strings receive no penalty

test_combined_penalties: Test that multiple penalties stack:

Create a string that triggers multiple detectors (long + high entropy + patterns)

Verify all penalties are applied

Verify final score is sum of base score minus all penalties

test_ranking_order: Create a mixed set of strings and verify:

High-quality strings (URLs, paths) rank higher than noise

Section weights influence ranking (strings from .rodata rank higher than .data)

Semantic tags boost scores appropriately

Noise penalties lower scores of garbage data

Final ranking order matches expected quality hierarchy

test_config_customization: Test that configuration changes affect behavior:

Create custom RankingConfig with different thresholds

Verify penalties change based on config

Test disabling noise penalties entirely

test_real_world_scenario: Use realistic binary string examples:

Mix of imports ("malloc", "printf"), URLs ("https://api.example.com"), paths ("/usr/bin/app"), and noise

Verify ranking produces sensible order

Use snapshot testing with insta to capture expected output

Each test should:

Create RankingEngine with appropriate config

Create test FoundString instances with realistic metadata (offset, section, tags)

Calculate scores or rank strings

Assert expected behavior with clear error messages

Include comments explaining what each test validates

Follow the pattern established in tests/integration_elf.rs for test structure and fixture handling.

📄 .kiro/specs/stringy-binary-analyzer/tasks.md ^(MODIFY) 🔗

Update the task tracking to reflect completion of task 11 (ranking system) and its subtasks. Find the section starting with - [ ] 11. Create ranking system foundation (around line 192) and update:

Task 11: Change checkbox from [ ] to [x] and update description to indicate completion.

Task 11.1: Change checkbox from [ ] to [x] - section weight scoring is implemented in RankingEngine::calculate_score.

Task 11.2: Change checkbox from [ ] to [x] - semantic boost scoring is implemented via tag_boosts in RankingConfig.

Task 11.3: Change checkbox from [ ] to [x] and add implementation details:

High entropy detection implemented in EntropyDetector

Excessive length penalty implemented in LengthDetector

Repeated pattern detection implemented in PatternDetector

Table data detection implemented in TableDataDetector

All integrated via NoiseDetector trait and RankingEngine

Comprehensive unit and integration tests included

Configurable thresholds via NoiseConfig

This tracks progress according to the project's task management system and provides visibility into what was implemented.

📄 README.md ^(MODIFY) 🔗

Update the development status section (around line 166-179) to reflect the completed ranking system implementation. Change the ranking system line from:

🚧 **Ranking System**: Section-aware scoring algorithm (framework in place)

To:

✅ **Ranking System**: Complete scoring with section weights, semantic boosts, and noise penalties

In the "Current Capabilities" section (around line 181-193), add a new bullet point after the symbol processing item:

Noise Detection: Intelligent filtering of low-quality strings using:

Shannon entropy calculation to detect random/encrypted data

Length-based penalties for padding and blob data

Pattern detection for repeated sequences

Table data heuristics for structured binary artifacts

Configurable thresholds for different analysis scenarios

This update reflects that the noise penalty detection feature is now implemented and highlights it as a key differentiator from standard strings command, supporting the "Eliminates noise" advantage mentioned in line 158.

Diagram

sequenceDiagram
    participant User
    participant RankingEngine
    participant NoiseDetector
    participant EntropyDetector
    participant LengthDetector
    participant PatternDetector
    participant TableDataDetector
    
    User->>RankingEngine: calculate_score(found_string, section_weight)
    RankingEngine->>RankingEngine: Convert section_weight to score
    RankingEngine->>RankingEngine: Add semantic boosts from tags
    
    loop For each NoiseDetector
        RankingEngine->>NoiseDetector: calculate_penalty(found_string)
        alt EntropyDetector
            NoiseDetector->>EntropyDetector: calculate_entropy(text)
            EntropyDetector-->>NoiseDetector: entropy value
            EntropyDetector-->>RankingEngine: penalty (0 to -30)
        else LengthDetector
            NoiseDetector->>LengthDetector: check length tiers
            LengthDetector-->>RankingEngine: penalty (0, -10, -20, or -30)
        else PatternDetector
            NoiseDetector->>PatternDetector: detect_repetition(text)
            PatternDetector-->>RankingEngine: penalty (0 or -25)
        else TableDataDetector
            NoiseDetector->>TableDataDetector: check delimiters & alphanum
            TableDataDetector-->>RankingEngine: penalty (0 or -20)
        end
        RankingEngine->>RankingEngine: Subtract penalty from score
    end
    
    RankingEngine-->>User: final_score

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a solid foundation with complete container parsers for ELF, PE, and Mach-O formats. The `FoundString` type already includes a `score: i32` field, but the ranking system doesn't exist yet. Section weights are already implemented in each parser (ranging from 1.0 to 10.0), providing a foundation for the scoring system. The classification and extraction modules are empty placeholders, and there's no ranking module yet. The project uses standard Rust testing with `insta` for snapshots and `criterion` for benchmarks.

### Approach

Implement a modular noise penalty detection system as part of the ranking infrastructure. Create the ranking module with a `NoiseDetector` trait for extensibility, then implement specific noise detectors (entropy, length, patterns, table data) as separate components. Integrate these into a `RankingEngine` that calculates final scores by combining section weights, semantic boosts, and noise penalties. Make thresholds configurable through a `RankingConfig` struct. Follow the existing codebase patterns: trait-based design, comprehensive unit tests, and clear separation of concerns.

### Reasoning

I explored the project structure and found that the core infrastructure (types, container parsers, section weighting) is complete. I examined the design document and requirements to understand the scoring formula: `Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty`. I reviewed the existing test patterns and confirmed that the ranking module needs to be created from scratch, with noise detection as one of its key components.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant RankingEngine
    participant NoiseDetector
    participant EntropyDetector
    participant LengthDetector
    participant PatternDetector
    participant TableDataDetector
    
    User->>RankingEngine: calculate_score(found_string, section_weight)
    RankingEngine->>RankingEngine: Convert section_weight to score
    RankingEngine->>RankingEngine: Add semantic boosts from tags
    
    loop For each NoiseDetector
        RankingEngine->>NoiseDetector: calculate_penalty(found_string)
        alt EntropyDetector
            NoiseDetector->>EntropyDetector: calculate_entropy(text)
            EntropyDetector-->>NoiseDetector: entropy value
            EntropyDetector-->>RankingEngine: penalty (0 to -30)
        else LengthDetector
            NoiseDetector->>LengthDetector: check length tiers
            LengthDetector-->>RankingEngine: penalty (0, -10, -20, or -30)
        else PatternDetector
            NoiseDetector->>PatternDetector: detect_repetition(text)
            PatternDetector-->>RankingEngine: penalty (0 or -25)
        else TableDataDetector
            NoiseDetector->>TableDataDetector: check delimiters & alphanum
            TableDataDetector-->>RankingEngine: penalty (0 or -20)
        end
        RankingEngine->>RankingEngine: Subtract penalty from score
    end
    
    RankingEngine-->>User: final_score

## Proposed File Changes

### src/classification/mod.rs(MODIFY)

References: 

- src/container/mod.rs

Add module declarations for the new ranking and noise detection modules. Export the `RankingEngine`, `RankingConfig`, and `NoiseDetector` trait for use by other parts of the codebase. This file currently only contains a comment, so add:

- `pub mod ranking;` - declares the ranking module
- `pub mod noise;` - declares the noise detection module
- Re-export key types: `pub use ranking::{RankingEngine, RankingConfig};` and `pub use noise::NoiseDetector;`

This follows the pattern established in `src/container/mod.rs` which declares submodules and re-exports parsers.

### src/classification/ranking.rs(NEW)

References: 

- src/types.rs
- .kiro/specs/stringy-binary-analyzer/design.md

Create the ranking system foundation with the following components:

**RankingConfig struct**: Configuration for ranking behavior with fields:
- `section_weight_multiplier: f32` (default 1.0) - scales section weights from container parsers
- `tag_boosts: HashMap<Tag, i32>` - semantic boost values per tag type (URL: +5, GUID: +5, registry: +5, filepath: +3, format string: +3, import/export: +2)
- `enable_noise_penalties: bool` (default true) - toggle noise detection
- `noise_config: NoiseConfig` - configuration for noise detectors

**NoiseConfig struct**: Nested configuration for noise detection thresholds:
- `entropy_threshold_low: f64` (default 4.0) - start of entropy penalty range
- `entropy_threshold_high: f64` (default 5.0) - maximum entropy before fixed penalty
- `entropy_penalty_max: i32` (default -30) - maximum entropy penalty
- `length_tier_1: usize` (default 256) - first length threshold
- `length_tier_2: usize` (default 512) - second length threshold
- `length_tier_3: usize` (default 1024) - third length threshold
- `length_penalty_tier_1: i32` (default -10)
- `length_penalty_tier_2: i32` (default -20)
- `length_penalty_tier_3: i32` (default -30)
- `pattern_repetition_threshold: f64` (default 0.1) - ratio below which string is considered repetitive
- `pattern_penalty: i32` (default -25)
- `table_delimiter_threshold: f64` (default 0.3) - delimiter ratio for table detection
- `table_alphanum_threshold: f64` (default 0.6) - alphanumeric ratio for table detection
- `table_penalty: i32` (default -20)

**RankingEngine struct**: Main scoring engine with fields:
- `config: RankingConfig`
- `noise_detectors: Vec<Box<dyn NoiseDetector>>` - pluggable noise detectors

**RankingEngine implementation**:
- `new(config: RankingConfig) -> Self` - constructor that initializes noise detectors based on config
- `calculate_score(&self, found_string: &FoundString, section_weight: f32) -> i32` - main scoring function that:
  1. Converts section weight to integer score (multiply by 10 to preserve precision)
  2. Adds semantic boosts based on tags
  3. Subtracts noise penalties from all enabled detectors
  4. Returns final score
- `rank_strings(&self, strings: &mut [FoundString], section_weights: &HashMap<String, f32>)` - sorts strings by score in descending order after calculating scores

**Default implementations**: Provide sensible defaults for `RankingConfig` and `NoiseConfig` using `Default` trait, matching the values specified in the issue.

Include comprehensive unit tests:
- Test score calculation with various combinations of section weights, tags, and noise penalties
- Test that noise penalties are correctly applied
- Test that configuration changes affect scoring
- Test ranking order with sample strings

### src/classification/noise.rs(NEW)

References: 

- src/types.rs

Create the noise detection module with the `NoiseDetector` trait and concrete implementations:

**NoiseDetector trait**: Define the interface for noise detection:
- `fn calculate_penalty(&self, string: &FoundString) -> i32` - returns penalty value (negative integer)
- `fn name(&self) -> &str` - returns detector name for debugging

**EntropyDetector struct**: Detects high-entropy (random-looking) strings:
- Fields: `threshold_low: f64`, `threshold_high: f64`, `max_penalty: i32`
- `new(threshold_low: f64, threshold_high: f64, max_penalty: i32) -> Self`
- `calculate_entropy(text: &str) -> f64` - private helper that:
  1. Counts character frequencies in the string
  2. Calculates Shannon entropy: sum of -p * log2(p) for each character probability
  3. Returns entropy value (0.0 to ~8.0 for printable ASCII)
- Implementation of `NoiseDetector::calculate_penalty`:
  - If entropy < threshold_low: return 0 (no penalty)
  - If entropy between thresholds: linear interpolation from 0 to max_penalty
  - If entropy > threshold_high: return max_penalty

**LengthDetector struct**: Penalizes excessively long strings:
- Fields: `tier_1: usize`, `tier_2: usize`, `tier_3: usize`, `penalty_1: i32`, `penalty_2: i32`, `penalty_3: i32`
- `new(tier_1: usize, tier_2: usize, tier_3: usize, penalty_1: i32, penalty_2: i32, penalty_3: i32) -> Self`
- Implementation of `NoiseDetector::calculate_penalty`:
  - Match on string length and return appropriate penalty based on tier
  - 0..=tier_1: 0 penalty
  - tier_1+1..=tier_2: penalty_1
  - tier_2+1..=tier_3: penalty_2
  - tier_3+1..: penalty_3

**PatternDetector struct**: Detects repeated character patterns:
- Fields: `repetition_threshold: f64`, `penalty: i32`
- `new(repetition_threshold: f64, penalty: i32) -> Self`
- `detect_single_char_repetition(text: &str) -> f64` - calculates ratio of unique chars to total length
- `is_repeating_pattern(text: &str, pattern_len: usize) -> bool` - checks if string is composed of repeating N-character pattern
- Implementation of `NoiseDetector::calculate_penalty`:
  - Check single character repetition ratio
  - If ratio < threshold: return penalty
  - Check for 2-4 character repeating patterns
  - If found: return penalty
  - Otherwise: return 0

**TableDataDetector struct**: Detects table-like structured data:
- Fields: `delimiter_threshold: f64`, `alphanum_threshold: f64`, `penalty: i32`
- `new(delimiter_threshold: f64, alphanum_threshold: f64, penalty: i32) -> Self`
- `calculate_delimiter_ratio(text: &str) -> f64` - counts delimiters (comma, pipe, tab, null) divided by length
- `calculate_alphanum_ratio(text: &str) -> f64` - counts alphanumeric chars divided by length
- Implementation of `NoiseDetector::calculate_penalty`:
  - Calculate both ratios
  - If delimiter_ratio > threshold OR alphanum_ratio < threshold: return penalty
  - Otherwise: return 0

**Helper function**: `create_default_detectors(config: &NoiseConfig) -> Vec<Box<dyn NoiseDetector>>` - factory function that creates all four detector instances with config values.

Include comprehensive unit tests for each detector:
- **EntropyDetector tests**: Test with low-entropy strings ("hello world"), medium-entropy strings ("H3ll0_W0r1d!"), high-entropy strings ("xK9#mP@vQ2%nR")
- **LengthDetector tests**: Test strings at each tier boundary (255, 256, 257, 511, 512, 513, 1023, 1024, 1025 chars)
- **PatternDetector tests**: Test single-char repetition ("AAAAAAA"), 2-char patterns ("ABABAB"), 3-char patterns ("123123123"), normal strings
- **TableDataDetector tests**: Test high-delimiter strings ("a,b,c,d,e,f"), low-alphanum strings ("!!!@@@###$$$"), normal strings
- Test that penalties are negative integers
- Test edge cases (empty strings, single characters, unicode)

### src/lib.rs(MODIFY)

Update the re-exports section to include the new ranking types. After the existing re-export block (around line 51-54), add:

```rust
pub use classification::{RankingEngine, RankingConfig, NoiseDetector};
```

This makes the ranking system accessible to users of the library without requiring them to import from nested modules. Update the module documentation comment (around line 40) to reflect that the classification module now includes ranking functionality:

Change:
- `classification`: Semantic analysis and tagging (🚧 Types defined)

To:
- `classification`: Semantic analysis, tagging, and ranking (🚧 Ranking system implemented)

This follows the existing pattern where commonly used types from submodules are re-exported at the library root level, as seen with the types module exports.

### Cargo.toml(MODIFY)

No changes needed. The existing dependencies are sufficient for the noise penalty detection implementation:

- Standard library provides `HashMap` for configuration and frequency counting
- No external crates needed for entropy calculation (using standard math)
- No external crates needed for pattern detection (using string iteration)

The implementation uses only standard Rust library features. If future enhancements require regex for more sophisticated pattern detection, the `regex` crate can be added later, but it's not needed for the current specification.

### tests/integration_ranking.rs(NEW)

References: 

- tests/integration_elf.rs
- src/types.rs

Create integration tests for the ranking system with noise penalty detection:

**Test fixtures**: Create sample `FoundString` instances representing different scenarios:
- High-quality strings: URLs, file paths, error messages with low entropy and reasonable length
- Noisy strings: high-entropy data, extremely long strings, repeated patterns, table data
- Mixed strings: legitimate strings with some noise characteristics

**Test cases**:

1. `test_entropy_penalty_integration`: Create strings with varying entropy levels and verify:
   - Low-entropy strings ("Hello World") receive no penalty
   - Medium-entropy strings receive proportional penalties
   - High-entropy strings ("xK9#mP@vQ2%nR") receive maximum penalty
   - Verify final scores reflect entropy differences

2. `test_length_penalty_integration`: Create strings at different length tiers:
   - Short strings (< 256 chars) receive no length penalty
   - Medium strings (256-512 chars) receive tier 1 penalty
   - Long strings (512-1024 chars) receive tier 2 penalty
   - Very long strings (> 1024 chars) receive tier 3 penalty
   - Verify scores decrease with length

3. `test_pattern_penalty_integration`: Test pattern detection:
   - Strings with >90% same character ("AAAAAAAAAA") receive pattern penalty
   - Strings with repeating 2-4 char patterns ("ABABABABAB") receive penalty
   - Normal strings with varied characters receive no penalty

4. `test_table_data_penalty_integration`: Test table detection:
   - Strings with high delimiter density ("a,b,c,d,e,f,g,h") receive penalty
   - Strings with low alphanumeric ratio ("!!!@@@###$$$") receive penalty
   - Normal strings receive no penalty

5. `test_combined_penalties`: Test that multiple penalties stack:
   - Create a string that triggers multiple detectors (long + high entropy + patterns)
   - Verify all penalties are applied
   - Verify final score is sum of base score minus all penalties

6. `test_ranking_order`: Create a mixed set of strings and verify:
   - High-quality strings (URLs, paths) rank higher than noise
   - Section weights influence ranking (strings from `.rodata` rank higher than `.data`)
   - Semantic tags boost scores appropriately
   - Noise penalties lower scores of garbage data
   - Final ranking order matches expected quality hierarchy

7. `test_config_customization`: Test that configuration changes affect behavior:
   - Create custom `RankingConfig` with different thresholds
   - Verify penalties change based on config
   - Test disabling noise penalties entirely

8. `test_real_world_scenario`: Use realistic binary string examples:
   - Mix of imports ("malloc", "printf"), URLs ("https://api.example.com"), paths ("/usr/bin/app"), and noise
   - Verify ranking produces sensible order
   - Use snapshot testing with `insta` to capture expected output

Each test should:
- Create `RankingEngine` with appropriate config
- Create test `FoundString` instances with realistic metadata (offset, section, tags)
- Calculate scores or rank strings
- Assert expected behavior with clear error messages
- Include comments explaining what each test validates

Follow the pattern established in `tests/integration_elf.rs` for test structure and fixture handling.

### .kiro/specs/stringy-binary-analyzer/tasks.md(MODIFY)

Update the task tracking to reflect completion of task 11 (ranking system) and its subtasks. Find the section starting with `- [ ] 11. Create ranking system foundation` (around line 192) and update:

**Task 11**: Change checkbox from `[ ]` to `[x]` and update description to indicate completion.

**Task 11.1**: Change checkbox from `[ ]` to `[x]` - section weight scoring is implemented in `RankingEngine::calculate_score`.

**Task 11.2**: Change checkbox from `[ ]` to `[x]` - semantic boost scoring is implemented via `tag_boosts` in `RankingConfig`.

**Task 11.3**: Change checkbox from `[ ]` to `[x]` and add implementation details:
- High entropy detection implemented in `EntropyDetector`
- Excessive length penalty implemented in `LengthDetector`
- Repeated pattern detection implemented in `PatternDetector`
- Table data detection implemented in `TableDataDetector`
- All integrated via `NoiseDetector` trait and `RankingEngine`
- Comprehensive unit and integration tests included
- Configurable thresholds via `NoiseConfig`

This tracks progress according to the project's task management system and provides visibility into what was implemented.

### README.md(MODIFY)

Update the development status section (around line 166-179) to reflect the completed ranking system implementation. Change the ranking system line from:

`🚧 **Ranking System**: Section-aware scoring algorithm (framework in place)`

To:

`✅ **Ranking System**: Complete scoring with section weights, semantic boosts, and noise penalties`

In the "Current Capabilities" section (around line 181-193), add a new bullet point after the symbol processing item:

- **Noise Detection**: Intelligent filtering of low-quality strings using:
  - Shannon entropy calculation to detect random/encrypted data
  - Length-based penalties for padding and blob data
  - Pattern detection for repeated sequences
  - Table data heuristics for structured binary artifacts
  - Configurable thresholds for different analysis scenarios

This update reflects that the noise penalty detection feature is now implemented and highlights it as a key differentiator from standard `strings` command, supporting the "Eliminates noise" advantage mentioned in line 158.

Developer Humor

🎯 Noise Penalty Haiku\n>\n> Random bytes beware,\n> Entropy reveals your truth—\n> Quality strings rise.\n>\n> 📊 Scoring with style!

Execution Information

Branch: main
Commit: bdd6206

💡 Tips

Supported Commands (Inside Comments)

Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Implemented: noise penalty detection (noise_penalty scoring) is live in the extraction/classification pipeline. Closing as completed.

Navigation Menu

Repository navigation

Implement Noise Penalty Detection for String Quality Scoring #24

Overview

Problem Statement

Proposed Solution

High Entropy Detection

Excessive Length Penalty

Repeated Pattern Detection

Table Data Heuristics

Implementation Plan

Acceptance Criteria

Technical Considerations

Dependencies

Requirements

Task ID

Related

✅ Enhanced Issue Title

✅ Comprehensive Issue Body

✅ Metadata

Plan

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement Noise Penalty Detection for String Quality Scoring #24

Description

Overview

Problem Statement

Proposed Solution

High Entropy Detection

Excessive Length Penalty

Repeated Pattern Detection

Table Data Heuristics

Implementation Plan

Acceptance Criteria

Technical Considerations

Dependencies

Requirements

Task ID

Related

Activity

unclesp1d3r commented on Nov 10, 2025

coderabbitai commented on Nov 10, 2025

✅ Enhanced Issue Title

✅ Comprehensive Issue Body

✅ Metadata

traycerai commented on Nov 10, 2025

Plan

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

unclesp1d3r commented on Jul 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions