Implement UTF-16BE String Extraction with Confidence Scoring and False Positive Prevention

Summary

Extend the UTF-16 string extractor to support big-endian byte order (UTF-16BE) with confidence scoring to distinguish legitimate UTF-16BE strings from binary data that coincidentally resembles UTF-16 encoding.

Background

While UTF-16LE (little-endian) is dominant in Windows PE binaries, UTF-16BE appears in:

Network protocol payloads and packet captures
Java .class files and serialized objects
Some file formats (certain PDFs, XML with BOM markers)
Cross-platform binaries targeting big-endian architectures (SPARC, PowerPC, some ARM)

Without UTF-16BE support, Stringy would miss meaningful strings in these binaries. However, naive UTF-16BE extraction can produce false positives since certain binary data patterns can resemble valid UTF-16BE sequences.

Problem Statement

The current extraction framework (src/extraction/mod.rs) lacks UTF-16BE support entirely. Additionally, both UTF-16LE and UTF-16BE extractors need confidence scoring to avoid extracting garbage that looks like valid UTF-16:

False Positive Scenarios:

Binary tables with null bytes in predictable positions
Numeric data that happens to have zeros in high/low byte positions
Padding or alignment bytes creating null-interleaved patterns
Compressed or encrypted data with statistical byte patterns

Requirements

Req 2.3: UTF-16BE extraction with byte-order awareness
Req 2.4: Confidence scoring for both UTF-16LE and UTF-16BE

Proposed Solution

1. Extend src/extraction/utf16.rs

Create a unified UTF-16 extractor that handles both byte orders:

pub struct Utf16Extractor {
    min_length: usize,
    confidence_threshold: f32,
}

impl Utf16Extractor {
    // Scan for UTF-16LE and UTF-16BE sequences
    pub fn extract(&self, data: &[u8], byte_order: ByteOrder) -> Vec<RawString>
    
    // Calculate confidence score (0.0 - 1.0)
    fn calculate_confidence(&self, chars: &[u16]) -> f32
}

2. Confidence Scoring Algorithm

Score UTF-16 candidates based on multiple heuristics:

Positive Indicators (+):

Valid Unicode code points (especially ASCII range U+0020-U+007E)
Common punctuation and whitespace
Consistent byte order throughout the string
Presence of word boundaries (spaces, newlines)
Character distribution matching natural language

Negative Indicators (-):

Invalid or private-use Unicode ranges
Excessive control characters (except \n, \r, \t)
Suspicious null byte patterns (every other byte, or repeating sequences)
High concentration of bytes outside printable ranges
Characters from mixed/incompatible Unicode blocks

Confidence Formula:

confidence = (valid_chars_weight × valid_ratio) 
           + (printable_weight × printable_ratio)
           + (language_weight × word_boundary_score)
           - (null_pattern_penalty)
           - (control_char_penalty)
           - (entropy_anomaly_penalty)

3. Null-Interleaved Pattern Detection

Implement specific checks for false-positive patterns:

Every-other-null: could be UTF-16LE "AB" or coincidental
Fixed-offset nulls: Nulls appearing at consistent intervals suggesting tables/structures
Run-length encoding artifacts: Repeated null sequences that aren't valid UTF-16

4. Integration Points

Wire into the main extraction pipeline (task 16)
Ensure proper handling of section boundaries
Support configurable confidence threshold via CLI (--utf16-confidence)
Add encoding detection heuristics to auto-select LE vs BE when unspecified

Implementation Checklist

Test Cases

Legitimate UTF-16BE: Java .class constant pool strings
False Positive: Null-interleaved numeric table data
Edge Case: Mixed ASCII and Unicode characters
Boundary: Minimum length UTF-16BE strings at section edges
Performance: Large binary with scattered null bytes

Dependencies

Blocked by: Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering #11 (UTF-16LE String Extraction) - Need base UTF-16 framework first
Related to: Implement Encoding Confidence Scoring and Integrate Section Weights into Ranking System #22 (Encoding Confidence Scoring and Ranking System)

Success Criteria

Correctly extracts UTF-16BE strings from Java .class files
Rejects false positives from binary data with <10% false positive rate
Confidence scores correlate with human assessment (validated against test corpus)
Performance impact: <20% overhead compared to basic UTF-16 scanning
Integration with ranking system properly weighs UTF-16BE confidence

Task ID

stringy-analyzer/utf16be-support-confidence-scoring

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

Plan

Observations

The codebase has a well-established extraction framework with ASCII extraction fully implemented, including confidence scoring via noise filters. UTF-16LE decoding exists in pe_resources.rs but only for resource extraction, not general section scanning. The Encoding enum already includes Utf16Le and Utf16Be variants. The ExtractionConfig has min_wide_length (default 3) ready for UTF-16 use. The BasicExtractor orchestrates extraction by calling format-specific extractors and applying noise filtering with configurable confidence thresholds.

Approach

Create a new utf16.rs module following the ASCII extractor pattern. Implement byte-order-aware scanning for both UTF-16LE and UTF-16BE with a ByteOrder enum (LE, BE, Auto). Add UTF-16-specific confidence scoring to detect false positives from null-interleaved binary data, then integrate with existing noise filters. Wire into BasicExtractor::extract_from_section to enable UTF-16 extraction alongside ASCII/UTF-8. Add comprehensive tests covering legitimate UTF-16 strings, false positive scenarios, and edge cases.

Reasoning

Listed the repository structure to understand the project layout. Read the extraction module files to understand the current architecture and patterns. Examined the ASCII extractor implementation to understand the extraction and confidence scoring approach. Reviewed the noise filtering framework to understand how confidence scores are calculated and combined. Checked the PE resources module to see existing UTF-16LE decoding logic. Searched for UTF-16 references to understand current usage. Reviewed integration tests to understand testing patterns and expectations.

Proposed File Changes

📄 src/extraction/utf16.rs ^(NEW) 🔗

References

📄 src/extraction/ascii.rs

📄 src/extraction/config.rs

📄 src/extraction/filters.rs

📄 src/types.rs

Create a new UTF-16 string extraction module following the pattern established in src/extraction/ascii.rs.

Module Structure:
ByteOrder enum: Define an enum with variants LE (little-endian), BE (big-endian), and Auto (detect both). This controls which byte orders to scan for.

Utf16ExtractionConfig struct: Create a configuration struct with fields:

min_length: usize - minimum string length in UTF-16 code units (default 3)

max_length: Option<usize> - maximum string length in UTF-16 code units

byte_order: ByteOrder - which byte order(s) to scan (default Auto)

confidence_threshold: f32 - minimum UTF-16-specific confidence (default 0.5)

Implement Default trait with sensible defaults

Add constructor new(min_length: usize) similar to ASCII config
UTF-16 confidence scoring functions:

calculate_utf16_confidence(chars: &[u16], byte_order: ByteOrder) -> f32 - main confidence calculator

check_valid_unicode_range(chars: &[u16]) -> f32 - validate code points are in valid Unicode ranges, penalize private use areas and invalid surrogates

check_null_pattern(chars: &[u16]) -> f32 - detect suspicious null patterns (every-other-null, fixed-offset nulls, excessive nulls)

check_ascii_ratio(chars: &[u16]) -> f32 - calculate ratio of ASCII-range characters (U+0020-U+007E), boost confidence for ASCII-heavy strings

check_printable_ratio(chars: &[u16]) -> f32 - ratio of printable characters including common Unicode ranges

check_byte_order_consistency(data: &[u8], byte_order: ByteOrder) -> f32 - verify byte order is consistent throughout

Combine these heuristics using weighted formula:
confidence = (valid_unicode_weight × valid_ratio)
           + (printable_weight × printable_ratio) 
           + (ascii_weight × ascii_ratio)
           - (null_pattern_penalty)
           - (invalid_range_penalty)
Clamp result to 0.0-1.0 range.
Core extraction functions:

extract_utf16_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - main extraction function that calls both LE and BE extractors based on config.byte_order

extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - scan for UTF-16LE sequences

extract_utf16be_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - scan for UTF-16BE sequences

extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString> - section-aware extraction with metadata population, similar to ascii::extract_from_section

Helper functions:

decode_utf16le(bytes: &[u8]) -> Result<(String, Vec<u16>)> - decode UTF-16LE bytes to String and return both the string and u16 vector for confidence scoring

decode_utf16be(bytes: &[u8]) -> Result<(String, Vec<u16>)> - decode UTF-16BE bytes to String and return both

is_valid_utf16_sequence(chars: &[u16]) -> bool - validate UTF-16 surrogate pairs and code points
Scanning Algorithm:
For each byte order:

Iterate through data in 2-byte chunks

Accumulate u16 values while they form valid UTF-16 sequences

When encountering invalid sequence or null terminator, check accumulated length

If length >= min_length, calculate UTF-16-specific confidence

If confidence >= threshold, decode to UTF-8 string

Create FoundString with appropriate encoding (Utf16Le or Utf16Be)

Handle end-of-buffer edge case

Confidence Scoring Details:

Valid Unicode ranges: U+0020-U+D7FF, U+E000-U+FFFD (exclude surrogates U+D800-U+DFFF, control chars U+0000-U+001F)

Null pattern detection: flag if >30% nulls, or nulls at regular intervals (every 2nd, 4th, 8th position)

ASCII ratio: boost confidence if >50% characters are in ASCII printable range

Byte order consistency: for Auto mode, ensure the detected byte order is consistent throughout the string

Integration with existing filters:
After UTF-16-specific confidence scoring, the decoded UTF-8 string should be passed through the existing CompositeNoiseFilter for additional validation. The final confidence should be the minimum of UTF-16 confidence and noise filter confidence, or a weighted combination.

Add comprehensive unit tests covering:

Basic UTF-16LE extraction

Basic UTF-16BE extraction

Auto byte order detection

Confidence scoring for legitimate strings

False positive detection (null-interleaved binary data)

Edge cases (minimum length, odd-length data, invalid surrogates)

Section-aware extraction with metadata

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

Integrate the new UTF-16 extraction module into the extraction framework.

Changes needed:
Module declaration (around line 99): Add pub mod utf16; after the existing module declarations.
Re-exports (around line 105): Add public re-exports for the UTF-16 module:
pub use utf16::{ByteOrder, Utf16ExtractionConfig, extract_utf16_strings, extract_from_section as extract_utf16_from_section};
ExtractionConfig struct (around lines 133-160): Add new fields for UTF-16 configuration:

utf16_byte_order: ByteOrder - which byte order(s) to scan (default Auto)

utf16_confidence_threshold: f32 - minimum UTF-16-specific confidence (default 0.5)

Update the Default impl to initialize these fields with sensible defaults.

ExtractionConfig::validate (around lines 185-212): Add validation for the new UTF-16 fields:

Ensure utf16_confidence_threshold is between 0.0 and 1.0

BasicExtractor::extract_from_section (around lines 416-537): Add UTF-16 extraction logic after the UTF-8 extraction block (around line 534):

Check if UTF-16LE or UTF-16BE is enabled in config.encodings or config.enabled_encodings

Create Utf16ExtractionConfig from the main config

Call utf16::extract_from_section with appropriate parameters

Merge results into found_strings vector

Apply the same noise filtering and confidence threshold logic as ASCII/UTF-8
Integration approach:
Follow the same pattern used for ASCII extraction:

Build UTF-16 config from extraction config

Call the UTF-16 extractor with section info

The UTF-16 extractor handles its own confidence scoring

Results are returned as FoundString with proper metadata

Optionally apply additional noise filtering if enabled

Encoding filtering:
Check both config.encodings and config.enabled_encodings for Encoding::Utf16Le and Encoding::Utf16Be to determine which byte orders to scan. If ByteOrder::Auto is configured, scan for both but only if at least one is enabled in the config.

📄 tests/test_utf16_extraction.rs ^(NEW) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

📄 tests/integration_extraction.rs ^(MODIFY)

Create comprehensive unit tests for UTF-16 extraction functionality.

Test categories:

Basic extraction tests:

test_extract_utf16le_basic - extract simple UTF-16LE strings ("Hello", "World")

test_extract_utf16be_basic - extract simple UTF-16BE strings

test_extract_utf16_auto_detects_le - Auto mode correctly detects LE

test_extract_utf16_auto_detects_be - Auto mode correctly detects BE

test_extract_utf16_mixed_ascii_unicode - strings with both ASCII and Unicode characters (e.g., "Hello 世界")

Minimum length filtering:

test_utf16_min_length_filtering - verify strings below min_length are filtered

test_utf16_exact_min_length - strings exactly at min_length are included

test_utf16_custom_min_length - test with custom min_length values

Confidence scoring tests:

test_utf16_confidence_legitimate_string - high confidence for normal text

test_utf16_confidence_ascii_heavy - boost for ASCII-heavy strings

test_utf16_confidence_null_pattern_penalty - low confidence for null-interleaved data

test_utf16_confidence_invalid_unicode - low confidence for invalid code points

test_utf16_confidence_threshold_filtering - strings below threshold are filtered

False positive prevention:

test_utf16_false_positive_binary_table - reject null-interleaved numeric table data

test_utf16_false_positive_every_other_null - reject patterns like "A\0B\0C\0" that could be ASCII or coincidental

test_utf16_false_positive_padding - reject padding bytes that look like UTF-16

test_utf16_false_positive_fixed_offset_nulls - reject structured binary data with nulls at regular intervals

Edge cases:

test_utf16_odd_length_data - handle odd-length byte arrays gracefully

test_utf16_invalid_surrogates - handle invalid surrogate pairs

test_utf16_string_at_buffer_start - string starting at offset 0

test_utf16_string_at_buffer_end - string ending at buffer boundary

test_utf16_empty_data - empty input returns empty result

test_utf16_max_length_filtering - strings exceeding max_length are filtered

Section-aware extraction:

test_utf16_extract_from_section_metadata - verify section name, offset, RVA are populated correctly

test_utf16_extract_from_section_offset_adjustment - absolute offsets calculated correctly

test_utf16_extract_from_section_rva_calculation - RVA calculated from section base + relative offset

test_utf16_extract_from_section_bounds_checking - handle section boundaries correctly

Real-world scenarios:

test_utf16_java_class_constant_pool - extract strings from Java .class file format (UTF-16BE)

test_utf16_windows_pe_strings - extract UTF-16LE strings from PE binary sections

test_utf16_mixed_encodings - data with both ASCII and UTF-16 strings

Byte order tests:

test_byte_order_le_only - ByteOrder::LE only scans little-endian

test_byte_order_be_only - ByteOrder::BE only scans big-endian

test_byte_order_auto_finds_both - ByteOrder::Auto finds both LE and BE strings

Test data examples:

UTF-16LE "Hello": [0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F, 0x00]

UTF-16BE "Hello": [0x00, 0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F]

False positive (null-interleaved): [0x41, 0x00, 0x42, 0x00, 0x43, 0x00] (could be "ABC" in ASCII or UTF-16LE)

Binary table: [0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04, 0x00] (numeric data)

Use the same testing patterns as src/extraction/ascii.rs tests for consistency.

📄 tests/integration_extraction.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Add integration tests for UTF-16 extraction within the existing extraction test suite.

New test functions to add:

test_basic_extractor_utf16le_strings - Test UTF-16LE extraction through BasicExtractor:

Create test data with embedded UTF-16LE strings

Create a SectionInfo with appropriate metadata

Configure ExtractionConfig to enable UTF-16LE encoding

Call extractor.extract_from_section

Verify extracted strings have correct encoding, text, offsets, and metadata

test_basic_extractor_utf16be_strings - Test UTF-16BE extraction:

Similar to UTF-16LE test but with big-endian byte order

Verify Encoding::Utf16Be is set correctly

test_basic_extractor_utf16_auto_detection - Test automatic byte order detection:

Create data with both UTF-16LE and UTF-16BE strings

Configure with ByteOrder::Auto

Verify both are extracted with correct encodings

test_basic_extractor_utf16_encoding_filtering - Test encoding selection:

Configure to only allow UTF-16LE (exclude UTF-16BE)

Verify only UTF-16LE strings are extracted

Test the reverse (only UTF-16BE)

test_basic_extractor_utf16_confidence_filtering - Test confidence threshold:

Create data with legitimate UTF-16 strings and false positives

Set a confidence threshold (e.g., 0.6)

Verify legitimate strings pass and false positives are filtered

test_basic_extractor_utf16_with_noise_filtering - Test integration with noise filters:

Enable noise filtering in config

Create UTF-16 strings with varying characteristics (normal text, repetitive, high entropy)

Verify noise filtering affects final confidence scores

test_basic_extractor_mixed_encodings - Test extraction of multiple encodings:

Create data with ASCII, UTF-8, UTF-16LE, and UTF-16BE strings

Enable all encodings in config

Verify all are extracted with correct encodings

Integration with existing tests:
Add these tests after the existing UTF-8 tests (around line 80) to maintain logical grouping. Follow the same patterns:

Use helper function get_fixture_path if testing with real binaries

Create SectionInfo with appropriate section types and weights

Use ExtractionConfig with explicit encoding configuration

Verify all FoundString fields are properly populated

Check confidence scores are reasonable

Configuration example:
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    enabled_encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    min_wide_length: 3,
    utf16_byte_order: ByteOrder::Auto,
    utf16_confidence_threshold: 0.5,
    noise_filtering_enabled: true,
    ..Default::default()
};

📄 src/extraction/pe_resources.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

Refactor the existing decode_utf16le function to be reusable and add UTF-16BE support.

Changes needed:

Move decode_utf16le to utf16.rs (optional but recommended): The decode_utf16le function (lines 372-390) could be moved to the new utf16.rs module and made public, then re-imported here. This promotes code reuse and keeps UTF-16 decoding logic centralized. If moved, update the function signature to return Result<String> and handle the null-trimming consistently.

Alternative - Keep as-is: If not moving the function, ensure the new utf16.rs module implements its own decoding functions that are consistent with this implementation. The key logic to maintain:

Handle odd-length input by truncating

Use u16::from_le_bytes for LE, u16::from_be_bytes for BE

Use String::from_utf16 for decoding

Trim trailing null terminators

Return appropriate errors for invalid UTF-16

Update decode_manifest_data function (lines 657-696): The UTF-16BE decoding logic (lines 687-696) is already implemented here. This can serve as a reference for the utf16.rs implementation. Ensure consistency in error handling and null-trimming behavior.

Documentation updates: Add a note in the module documentation (around line 1) mentioning that general UTF-16 extraction is now handled by the utf16 module, while this module focuses on PE-specific resource extraction.

No breaking changes: The existing functions should continue to work as-is. The new utf16.rs module will provide general-purpose UTF-16 extraction, while this module remains focused on PE resource-specific extraction.

📄 docs/src/string-extraction.md ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update the string extraction documentation to include UTF-16 support.

Sections to update:
Supported Encodings section: Add UTF-16LE and UTF-16BE to the list of supported encodings:

UTF-16LE (Little-Endian): Common in Windows PE binaries, default 3 character minimum

UTF-16BE (Big-Endian): Found in Java .class files, network protocols, some cross-platform binaries

Automatic byte order detection available

Confidence Scoring section: Add a subsection on UTF-16-specific confidence scoring:

Explain the false positive problem with null-interleaved binary data

Describe the heuristics used: valid Unicode ranges, null pattern detection, ASCII ratio, printable character ratio

Mention the configurable confidence threshold

Explain how UTF-16 confidence combines with general noise filtering

Configuration section: Document the new UTF-16 configuration options:

min_wide_length - minimum length for UTF-16 strings (default 3)

utf16_byte_order - which byte order(s) to scan (LE, BE, Auto)

utf16_confidence_threshold - minimum UTF-16-specific confidence (default 0.5)

encodings - include Utf16Le and/or Utf16Be to enable UTF-16 extraction
Examples section: Add examples of UTF-16 extraction:
// Extract UTF-16LE strings from Windows PE binary
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le],
    min_wide_length: 3,
    utf16_confidence_threshold: 0.6,
    ..Default::default()
};

// Extract both UTF-16LE and UTF-16BE with auto-detection
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    utf16_byte_order: ByteOrder::Auto,
    ..Default::default()
};
False Positive Prevention section: Add a new section explaining:

Why UTF-16 extraction is prone to false positives

Examples of binary data that can look like UTF-16 (null-interleaved tables, padding)

How the confidence scoring system mitigates this

Recommendations for tuning the confidence threshold

Performance Considerations section: Mention that UTF-16 scanning adds overhead:

Scanning for both byte orders doubles the work

Confidence scoring adds computational cost

Recommend using specific byte order (LE or BE) when the target format is known

Auto mode is best for unknown or mixed-format binaries

Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module
    participant ConfidenceScorer
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if UTF-16 enabled in config
    
    alt UTF-16LE or UTF-16BE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, utf16_config)
        
        alt ByteOrder::Auto
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        else ByteOrder::LE
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
        else ByteOrder::BE
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        end
        
        loop For each candidate string
            Utf16Module->>ConfidenceScorer: calculate_utf16_confidence(chars, byte_order)
            ConfidenceScorer->>ConfidenceScorer: check_valid_unicode_range()
            ConfidenceScorer->>ConfidenceScorer: check_null_pattern()
            ConfidenceScorer->>ConfidenceScorer: check_ascii_ratio()
            ConfidenceScorer->>ConfidenceScorer: check_printable_ratio()
            ConfidenceScorer-->>Utf16Module: utf16_confidence (0.0-1.0)
            
            alt utf16_confidence >= threshold
                Utf16Module->>Utf16Module: decode_to_utf8(chars)
                
                opt Noise filtering enabled
                    Utf16Module->>NoiseFilter: calculate_confidence(text, context)
                    NoiseFilter-->>Utf16Module: noise_confidence (0.0-1.0)
                    Utf16Module->>Utf16Module: final_confidence = min(utf16_conf, noise_conf)
                end
                
                alt final_confidence >= min_threshold
                    Utf16Module->>FoundString: Create with metadata
                    FoundString-->>Utf16Module: found_string
                end
            end
        end
        
        Utf16Module-->>BasicExtractor: Vec<FoundString>
        BasicExtractor->>BasicExtractor: Merge with ASCII/UTF-8 results
    end
    
    BasicExtractor-->>User: Vec<FoundString>

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established extraction framework with ASCII extraction fully implemented, including confidence scoring via noise filters. UTF-16LE decoding exists in `pe_resources.rs` but only for resource extraction, not general section scanning. The `Encoding` enum already includes `Utf16Le` and `Utf16Be` variants. The `ExtractionConfig` has `min_wide_length` (default 3) ready for UTF-16 use. The `BasicExtractor` orchestrates extraction by calling format-specific extractors and applying noise filtering with configurable confidence thresholds.

### Approach

Create a new `utf16.rs` module following the ASCII extractor pattern. Implement byte-order-aware scanning for both UTF-16LE and UTF-16BE with a `ByteOrder` enum (LE, BE, Auto). Add UTF-16-specific confidence scoring to detect false positives from null-interleaved binary data, then integrate with existing noise filters. Wire into `BasicExtractor::extract_from_section` to enable UTF-16 extraction alongside ASCII/UTF-8. Add comprehensive tests covering legitimate UTF-16 strings, false positive scenarios, and edge cases.

### Reasoning

Listed the repository structure to understand the project layout. Read the extraction module files to understand the current architecture and patterns. Examined the ASCII extractor implementation to understand the extraction and confidence scoring approach. Reviewed the noise filtering framework to understand how confidence scores are calculated and combined. Checked the PE resources module to see existing UTF-16LE decoding logic. Searched for UTF-16 references to understand current usage. Reviewed integration tests to understand testing patterns and expectations.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module
    participant ConfidenceScorer
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if UTF-16 enabled in config
    
    alt UTF-16LE or UTF-16BE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, utf16_config)
        
        alt ByteOrder::Auto
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        else ByteOrder::LE
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
        else ByteOrder::BE
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        end
        
        loop For each candidate string
            Utf16Module->>ConfidenceScorer: calculate_utf16_confidence(chars, byte_order)
            ConfidenceScorer->>ConfidenceScorer: check_valid_unicode_range()
            ConfidenceScorer->>ConfidenceScorer: check_null_pattern()
            ConfidenceScorer->>ConfidenceScorer: check_ascii_ratio()
            ConfidenceScorer->>ConfidenceScorer: check_printable_ratio()
            ConfidenceScorer-->>Utf16Module: utf16_confidence (0.0-1.0)
            
            alt utf16_confidence >= threshold
                Utf16Module->>Utf16Module: decode_to_utf8(chars)
                
                opt Noise filtering enabled
                    Utf16Module->>NoiseFilter: calculate_confidence(text, context)
                    NoiseFilter-->>Utf16Module: noise_confidence (0.0-1.0)
                    Utf16Module->>Utf16Module: final_confidence = min(utf16_conf, noise_conf)
                end
                
                alt final_confidence >= min_threshold
                    Utf16Module->>FoundString: Create with metadata
                    FoundString-->>Utf16Module: found_string
                end
            end
        end
        
        Utf16Module-->>BasicExtractor: Vec<FoundString>
        BasicExtractor->>BasicExtractor: Merge with ASCII/UTF-8 results
    end
    
    BasicExtractor-->>User: Vec<FoundString>

## Proposed File Changes

### src/extraction/utf16.rs(NEW)

References: 

- src/extraction/ascii.rs
- src/extraction/config.rs
- src/extraction/filters.rs
- src/types.rs

Create a new UTF-16 string extraction module following the pattern established in `src/extraction/ascii.rs`.

**Module Structure:**

1. **ByteOrder enum:** Define an enum with variants `LE` (little-endian), `BE` (big-endian), and `Auto` (detect both). This controls which byte orders to scan for.

2. **Utf16ExtractionConfig struct:** Create a configuration struct with fields:
   - `min_length: usize` - minimum string length in UTF-16 code units (default 3)
   - `max_length: Option<usize>` - maximum string length in UTF-16 code units
   - `byte_order: ByteOrder` - which byte order(s) to scan (default Auto)
   - `confidence_threshold: f32` - minimum UTF-16-specific confidence (default 0.5)
   - Implement `Default` trait with sensible defaults
   - Add constructor `new(min_length: usize)` similar to ASCII config

3. **UTF-16 confidence scoring functions:**
   - `calculate_utf16_confidence(chars: &[u16], byte_order: ByteOrder) -> f32` - main confidence calculator
   - `check_valid_unicode_range(chars: &[u16]) -> f32` - validate code points are in valid Unicode ranges, penalize private use areas and invalid surrogates
   - `check_null_pattern(chars: &[u16]) -> f32` - detect suspicious null patterns (every-other-null, fixed-offset nulls, excessive nulls)
   - `check_ascii_ratio(chars: &[u16]) -> f32` - calculate ratio of ASCII-range characters (U+0020-U+007E), boost confidence for ASCII-heavy strings
   - `check_printable_ratio(chars: &[u16]) -> f32` - ratio of printable characters including common Unicode ranges
   - `check_byte_order_consistency(data: &[u8], byte_order: ByteOrder) -> f32` - verify byte order is consistent throughout
   
   Combine these heuristics using weighted formula:
   ```
   confidence = (valid_unicode_weight × valid_ratio)
              + (printable_weight × printable_ratio) 
              + (ascii_weight × ascii_ratio)
              - (null_pattern_penalty)
              - (invalid_range_penalty)
   ```
   Clamp result to 0.0-1.0 range.

4. **Core extraction functions:**
   - `extract_utf16_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - main extraction function that calls both LE and BE extractors based on config.byte_order
   - `extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - scan for UTF-16LE sequences
   - `extract_utf16be_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - scan for UTF-16BE sequences
   - `extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>` - section-aware extraction with metadata population, similar to `ascii::extract_from_section`

5. **Helper functions:**
   - `decode_utf16le(bytes: &[u8]) -> Result<(String, Vec<u16>)>` - decode UTF-16LE bytes to String and return both the string and u16 vector for confidence scoring
   - `decode_utf16be(bytes: &[u8]) -> Result<(String, Vec<u16>)>` - decode UTF-16BE bytes to String and return both
   - `is_valid_utf16_sequence(chars: &[u16]) -> bool` - validate UTF-16 surrogate pairs and code points

**Scanning Algorithm:**
For each byte order:
1. Iterate through data in 2-byte chunks
2. Accumulate u16 values while they form valid UTF-16 sequences
3. When encountering invalid sequence or null terminator, check accumulated length
4. If length >= min_length, calculate UTF-16-specific confidence
5. If confidence >= threshold, decode to UTF-8 string
6. Create FoundString with appropriate encoding (Utf16Le or Utf16Be)
7. Handle end-of-buffer edge case

**Confidence Scoring Details:**
- Valid Unicode ranges: U+0020-U+D7FF, U+E000-U+FFFD (exclude surrogates U+D800-U+DFFF, control chars U+0000-U+001F)
- Null pattern detection: flag if >30% nulls, or nulls at regular intervals (every 2nd, 4th, 8th position)
- ASCII ratio: boost confidence if >50% characters are in ASCII printable range
- Byte order consistency: for Auto mode, ensure the detected byte order is consistent throughout the string

**Integration with existing filters:**
After UTF-16-specific confidence scoring, the decoded UTF-8 string should be passed through the existing `CompositeNoiseFilter` for additional validation. The final confidence should be the minimum of UTF-16 confidence and noise filter confidence, or a weighted combination.

Add comprehensive unit tests covering:
- Basic UTF-16LE extraction
- Basic UTF-16BE extraction
- Auto byte order detection
- Confidence scoring for legitimate strings
- False positive detection (null-interleaved binary data)
- Edge cases (minimum length, odd-length data, invalid surrogates)
- Section-aware extraction with metadata

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs

Integrate the new UTF-16 extraction module into the extraction framework.

**Changes needed:**

1. **Module declaration (around line 99):** Add `pub mod utf16;` after the existing module declarations.

2. **Re-exports (around line 105):** Add public re-exports for the UTF-16 module:
   ```rust
   pub use utf16::{ByteOrder, Utf16ExtractionConfig, extract_utf16_strings, extract_from_section as extract_utf16_from_section};
   ```

3. **ExtractionConfig struct (around lines 133-160):** Add new fields for UTF-16 configuration:
   - `utf16_byte_order: ByteOrder` - which byte order(s) to scan (default Auto)
   - `utf16_confidence_threshold: f32` - minimum UTF-16-specific confidence (default 0.5)
   
   Update the `Default` impl to initialize these fields with sensible defaults.

4. **ExtractionConfig::validate (around lines 185-212):** Add validation for the new UTF-16 fields:
   - Ensure `utf16_confidence_threshold` is between 0.0 and 1.0

5. **BasicExtractor::extract_from_section (around lines 416-537):** Add UTF-16 extraction logic after the UTF-8 extraction block (around line 534):
   - Check if UTF-16LE or UTF-16BE is enabled in `config.encodings` or `config.enabled_encodings`
   - Create `Utf16ExtractionConfig` from the main config
   - Call `utf16::extract_from_section` with appropriate parameters
   - Merge results into `found_strings` vector
   - Apply the same noise filtering and confidence threshold logic as ASCII/UTF-8

**Integration approach:**
Follow the same pattern used for ASCII extraction:
1. Build UTF-16 config from extraction config
2. Call the UTF-16 extractor with section info
3. The UTF-16 extractor handles its own confidence scoring
4. Results are returned as `FoundString` with proper metadata
5. Optionally apply additional noise filtering if enabled

**Encoding filtering:**
Check both `config.encodings` and `config.enabled_encodings` for `Encoding::Utf16Le` and `Encoding::Utf16Be` to determine which byte orders to scan. If `ByteOrder::Auto` is configured, scan for both but only if at least one is enabled in the config.

### tests/test_utf16_extraction.rs(NEW)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs
- tests/integration_extraction.rs(MODIFY)

Create comprehensive unit tests for UTF-16 extraction functionality.

**Test categories:**

1. **Basic extraction tests:**
   - `test_extract_utf16le_basic` - extract simple UTF-16LE strings ("Hello", "World")
   - `test_extract_utf16be_basic` - extract simple UTF-16BE strings
   - `test_extract_utf16_auto_detects_le` - Auto mode correctly detects LE
   - `test_extract_utf16_auto_detects_be` - Auto mode correctly detects BE
   - `test_extract_utf16_mixed_ascii_unicode` - strings with both ASCII and Unicode characters (e.g., "Hello 世界")

2. **Minimum length filtering:**
   - `test_utf16_min_length_filtering` - verify strings below min_length are filtered
   - `test_utf16_exact_min_length` - strings exactly at min_length are included
   - `test_utf16_custom_min_length` - test with custom min_length values

3. **Confidence scoring tests:**
   - `test_utf16_confidence_legitimate_string` - high confidence for normal text
   - `test_utf16_confidence_ascii_heavy` - boost for ASCII-heavy strings
   - `test_utf16_confidence_null_pattern_penalty` - low confidence for null-interleaved data
   - `test_utf16_confidence_invalid_unicode` - low confidence for invalid code points
   - `test_utf16_confidence_threshold_filtering` - strings below threshold are filtered

4. **False positive prevention:**
   - `test_utf16_false_positive_binary_table` - reject null-interleaved numeric table data
   - `test_utf16_false_positive_every_other_null` - reject patterns like "A\0B\0C\0" that could be ASCII or coincidental
   - `test_utf16_false_positive_padding` - reject padding bytes that look like UTF-16
   - `test_utf16_false_positive_fixed_offset_nulls` - reject structured binary data with nulls at regular intervals

5. **Edge cases:**
   - `test_utf16_odd_length_data` - handle odd-length byte arrays gracefully
   - `test_utf16_invalid_surrogates` - handle invalid surrogate pairs
   - `test_utf16_string_at_buffer_start` - string starting at offset 0
   - `test_utf16_string_at_buffer_end` - string ending at buffer boundary
   - `test_utf16_empty_data` - empty input returns empty result
   - `test_utf16_max_length_filtering` - strings exceeding max_length are filtered

6. **Section-aware extraction:**
   - `test_utf16_extract_from_section_metadata` - verify section name, offset, RVA are populated correctly
   - `test_utf16_extract_from_section_offset_adjustment` - absolute offsets calculated correctly
   - `test_utf16_extract_from_section_rva_calculation` - RVA calculated from section base + relative offset
   - `test_utf16_extract_from_section_bounds_checking` - handle section boundaries correctly

7. **Real-world scenarios:**
   - `test_utf16_java_class_constant_pool` - extract strings from Java .class file format (UTF-16BE)
   - `test_utf16_windows_pe_strings` - extract UTF-16LE strings from PE binary sections
   - `test_utf16_mixed_encodings` - data with both ASCII and UTF-16 strings

8. **Byte order tests:**
   - `test_byte_order_le_only` - ByteOrder::LE only scans little-endian
   - `test_byte_order_be_only` - ByteOrder::BE only scans big-endian  
   - `test_byte_order_auto_finds_both` - ByteOrder::Auto finds both LE and BE strings

**Test data examples:**
- UTF-16LE "Hello": `[0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F, 0x00]`
- UTF-16BE "Hello": `[0x00, 0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F]`
- False positive (null-interleaved): `[0x41, 0x00, 0x42, 0x00, 0x43, 0x00]` (could be "ABC" in ASCII or UTF-16LE)
- Binary table: `[0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04, 0x00]` (numeric data)

Use the same testing patterns as `src/extraction/ascii.rs` tests for consistency.

### tests/integration_extraction.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Add integration tests for UTF-16 extraction within the existing extraction test suite.

**New test functions to add:**

1. **test_basic_extractor_utf16le_strings** - Test UTF-16LE extraction through BasicExtractor:
   - Create test data with embedded UTF-16LE strings
   - Create a SectionInfo with appropriate metadata
   - Configure ExtractionConfig to enable UTF-16LE encoding
   - Call `extractor.extract_from_section`
   - Verify extracted strings have correct encoding, text, offsets, and metadata

2. **test_basic_extractor_utf16be_strings** - Test UTF-16BE extraction:
   - Similar to UTF-16LE test but with big-endian byte order
   - Verify `Encoding::Utf16Be` is set correctly

3. **test_basic_extractor_utf16_auto_detection** - Test automatic byte order detection:
   - Create data with both UTF-16LE and UTF-16BE strings
   - Configure with `ByteOrder::Auto`
   - Verify both are extracted with correct encodings

4. **test_basic_extractor_utf16_encoding_filtering** - Test encoding selection:
   - Configure to only allow UTF-16LE (exclude UTF-16BE)
   - Verify only UTF-16LE strings are extracted
   - Test the reverse (only UTF-16BE)

5. **test_basic_extractor_utf16_confidence_filtering** - Test confidence threshold:
   - Create data with legitimate UTF-16 strings and false positives
   - Set a confidence threshold (e.g., 0.6)
   - Verify legitimate strings pass and false positives are filtered

6. **test_basic_extractor_utf16_with_noise_filtering** - Test integration with noise filters:
   - Enable noise filtering in config
   - Create UTF-16 strings with varying characteristics (normal text, repetitive, high entropy)
   - Verify noise filtering affects final confidence scores

7. **test_basic_extractor_mixed_encodings** - Test extraction of multiple encodings:
   - Create data with ASCII, UTF-8, UTF-16LE, and UTF-16BE strings
   - Enable all encodings in config
   - Verify all are extracted with correct encodings

**Integration with existing tests:**
Add these tests after the existing UTF-8 tests (around line 80) to maintain logical grouping. Follow the same patterns:
- Use helper function `get_fixture_path` if testing with real binaries
- Create `SectionInfo` with appropriate section types and weights
- Use `ExtractionConfig` with explicit encoding configuration
- Verify all `FoundString` fields are properly populated
- Check confidence scores are reasonable

**Configuration example:**
```rust
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    enabled_encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    min_wide_length: 3,
    utf16_byte_order: ByteOrder::Auto,
    utf16_confidence_threshold: 0.5,
    noise_filtering_enabled: true,
    ..Default::default()
};
```

### src/extraction/pe_resources.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)

Refactor the existing `decode_utf16le` function to be reusable and add UTF-16BE support.

**Changes needed:**

1. **Move decode_utf16le to utf16.rs (optional but recommended):** The `decode_utf16le` function (lines 372-390) could be moved to the new `utf16.rs` module and made public, then re-imported here. This promotes code reuse and keeps UTF-16 decoding logic centralized. If moved, update the function signature to return `Result<String>` and handle the null-trimming consistently.

2. **Alternative - Keep as-is:** If not moving the function, ensure the new `utf16.rs` module implements its own decoding functions that are consistent with this implementation. The key logic to maintain:
   - Handle odd-length input by truncating
   - Use `u16::from_le_bytes` for LE, `u16::from_be_bytes` for BE
   - Use `String::from_utf16` for decoding
   - Trim trailing null terminators
   - Return appropriate errors for invalid UTF-16

3. **Update decode_manifest_data function (lines 657-696):** The UTF-16BE decoding logic (lines 687-696) is already implemented here. This can serve as a reference for the `utf16.rs` implementation. Ensure consistency in error handling and null-trimming behavior.

4. **Documentation updates:** Add a note in the module documentation (around line 1) mentioning that general UTF-16 extraction is now handled by the `utf16` module, while this module focuses on PE-specific resource extraction.

**No breaking changes:** The existing functions should continue to work as-is. The new `utf16.rs` module will provide general-purpose UTF-16 extraction, while this module remains focused on PE resource-specific extraction.

### docs/src/string-extraction.md(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update the string extraction documentation to include UTF-16 support.

**Sections to update:**

1. **Supported Encodings section:** Add UTF-16LE and UTF-16BE to the list of supported encodings:
   - UTF-16LE (Little-Endian): Common in Windows PE binaries, default 3 character minimum
   - UTF-16BE (Big-Endian): Found in Java .class files, network protocols, some cross-platform binaries
   - Automatic byte order detection available

2. **Confidence Scoring section:** Add a subsection on UTF-16-specific confidence scoring:
   - Explain the false positive problem with null-interleaved binary data
   - Describe the heuristics used: valid Unicode ranges, null pattern detection, ASCII ratio, printable character ratio
   - Mention the configurable confidence threshold
   - Explain how UTF-16 confidence combines with general noise filtering

3. **Configuration section:** Document the new UTF-16 configuration options:
   - `min_wide_length` - minimum length for UTF-16 strings (default 3)
   - `utf16_byte_order` - which byte order(s) to scan (LE, BE, Auto)
   - `utf16_confidence_threshold` - minimum UTF-16-specific confidence (default 0.5)
   - `encodings` - include `Utf16Le` and/or `Utf16Be` to enable UTF-16 extraction

4. **Examples section:** Add examples of UTF-16 extraction:
   ```rust
   // Extract UTF-16LE strings from Windows PE binary
   let config = ExtractionConfig {
       encodings: vec![Encoding::Utf16Le],
       min_wide_length: 3,
       utf16_confidence_threshold: 0.6,
       ..Default::default()
   };
   
   // Extract both UTF-16LE and UTF-16BE with auto-detection
   let config = ExtractionConfig {
       encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
       utf16_byte_order: ByteOrder::Auto,
       ..Default::default()
   };
   ```

5. **False Positive Prevention section:** Add a new section explaining:
   - Why UTF-16 extraction is prone to false positives
   - Examples of binary data that can look like UTF-16 (null-interleaved tables, padding)
   - How the confidence scoring system mitigates this
   - Recommendations for tuning the confidence threshold

6. **Performance Considerations section:** Mention that UTF-16 scanning adds overhead:
   - Scanning for both byte orders doubles the work
   - Confidence scoring adds computational cost
   - Recommend using specific byte order (LE or BE) when the target format is known
   - Auto mode is best for unknown or mixed-format binaries

Developer Humor

🎭 The UTF-16 Extractor's Tale\n>\n> In a world of bytes, both big and small,\n> Where endianness confounds us all,\n> A string extractor, brave and true,\n> Scans for patterns, old and new.\n>\n> \Is it LE or BE?\ it cries,\n> \Or just null bytes in disguise?\n> With confidence scores, it takes its stand,\n> Rejecting noise with a steady hand! 🔍✨

Execution Information

Branch: main
Commit: c50e9b6

💡 Tips

Supported Commands (Inside Comments)

Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

@unclesp1d3r

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch current issue details
gh issue view 12 --json title,body,labels,assignees,milestone,state --repo EvilBit-Labs/StringyMcStringFace

Length of output: 1334

🏁 Script executed:

#!/bin/bash
# Search for UTF-16 related code and extractors
fd -e py -e rs -e go -e java | head -20

Length of output: 277

🏁 Script executed:

#!/bin/bash
# Look for existing UTF-16 implementation
rg -n "UTF-?16" --type-add 'code:*.{py,rs,go,java,cpp,c}' --type code -C 3

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Find extractor-related files
fd -t f extractor

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Look at the project structure
ls -la

Length of output: 1765

🏁 Script executed:

#!/bin/bash
# Look at the extraction module
cat src/extraction/mod.rs

Length of output: 26

🏁 Script executed:

#!/bin/bash
# Search for string-related code
rg -n "string|String" src/extraction/mod.rs -C 2

Length of output: 28

🏁 Script executed:

#!/bin/bash
# Check README for project context
cat README.md

Length of output: 7196

🏁 Script executed:

#!/bin/bash
# Look for any encoding or UTF references in Rust files
rg -n "encoding|utf|UTF|string_extraction" --type rust

Length of output: 422

🏁 Script executed:

#!/bin/bash
# Check the types file for understanding data structures
cat src/types.rs

Length of output: 5122

🏁 Script executed:

#!/bin/bash
# Check concept.md for more context on UTF-16 handling
cat concept.md

Length of output: 6081

🏁 Script executed:

#!/bin/bash
# Look for related issues about UTF-16LE
gh issue list --repo EvilBit-Labs/StringyMcStringFace --search "UTF-16" --json number,title,state,labels

Length of output: 11166

🏁 Script executed:

#!/bin/bash
# Check the tasks.md for implementation details
cat .kiro/specs/stringy-binary-analyzer/tasks.md

Length of output: 11157

🏁 Script executed:

#!/bin/bash
# List available labels in the repository
gh label list --repo EvilBit-Labs/StringyMcStringFace --json name,description,color --limit 100

Length of output: 2427

🏁 Script executed:

#!/bin/bash
# Check for confidence scoring or false positive related code/comments
rg -n "confidence|false.positive|score" --type rust -C 2

Length of output: 702

🏁 Script executed:

#!/bin/bash
# Update issue #12 with enhanced content
gh issue edit 12 --repo EvilBit-Labs/StringyMcStringFace \
  --title "Implement UTF-16BE String Extraction with Confidence Scoring and False Positive Prevention" \
  --body "## Summary

Extend the UTF-16 string extractor to support big-endian byte order (UTF-16BE) with confidence scoring to distinguish legitimate UTF-16BE strings from binary data that coincidentally resembles UTF-16 encoding.

## Background

While UTF-16LE (little-endian) is dominant in Windows PE binaries, UTF-16BE appears in:
- Network protocol payloads and packet captures
- Java .class files and serialized objects  
- Some file formats (certain PDFs, XML with BOM markers)
- Cross-platform binaries targeting big-endian architectures (SPARC, PowerPC, some ARM)

Without UTF-16BE support, Stringy would miss meaningful strings in these binaries. However, naive UTF-16BE extraction can produce false positives since certain binary data patterns can resemble valid UTF-16BE sequences.

## Problem Statement

The current extraction framework (src/extraction/mod.rs) lacks UTF-16BE support entirely. Additionally, both UTF-16LE and UTF-16BE extractors need confidence scoring to avoid extracting garbage that looks like valid UTF-16:

**False Positive Scenarios:**
1. Binary tables with null bytes in predictable positions
2. Numeric data that happens to have zeros in high/low byte positions
3. Padding or alignment bytes creating null-interleaved patterns
4. Compressed or encrypted data with statistical byte patterns

## Requirements

- **Req 2.3:** UTF-16BE extraction with byte-order awareness
- **Req 2.4:** Confidence scoring for both UTF-16LE and UTF-16BE

## Proposed Solution

### 1. Extend src/extraction/utf16.rs

Create a unified UTF-16 extractor that handles both byte orders:

\`\`\`rust
pub struct Utf16Extractor {
    min_length: usize,
    confidence_threshold: f32,
}

impl Utf16Extractor {
    // Scan for UTF-16LE and UTF-16BE sequences
    pub fn extract(&self, data: &[u8], byte_order: ByteOrder) -> Vec<RawString>
    
    // Calculate confidence score (0.0 - 1.0)
    fn calculate_confidence(&self, chars: &[u16]) -> f32
}
\`\`\`

### 2. Confidence Scoring Algorithm

Score UTF-16 candidates based on multiple heuristics:

**Positive Indicators (+):**
- Valid Unicode code points (especially ASCII range U+0020-U+007E)
- Common punctuation and whitespace
- Consistent byte order throughout the string
- Presence of word boundaries (spaces, newlines)
- Character distribution matching natural language

**Negative Indicators (-):**
- Invalid or private-use Unicode ranges
- Excessive control characters (except \\n, \\r, \\t)
- Suspicious null byte patterns (every other byte, or repeating sequences)
- High concentration of bytes outside printable ranges
- Characters from mixed/incompatible Unicode blocks

**Confidence Formula:**
\`\`\`
confidence = (valid_chars_weight × valid_ratio) 
           + (printable_weight × printable_ratio)
           + (language_weight × word_boundary_score)
           - (null_pattern_penalty)
           - (control_char_penalty)
           - (entropy_anomaly_penalty)
\`\`\`

### 3. Null-Interleaved Pattern Detection

Implement specific checks for false-positive patterns:

- **Every-other-null:** `[0x41, 0x00, 0x42, 0x00]` could be UTF-16LE \"AB\" or coincidental
- **Fixed-offset nulls:** Nulls appearing at consistent intervals suggesting tables/structures
- **Run-length encoding artifacts:** Repeated null sequences that aren't valid UTF-16

### 4. Integration Points

- Wire into the main extraction pipeline (task 16)
- Ensure proper handling of section boundaries
- Support configurable confidence threshold via CLI (--utf16-confidence)
- Add encoding detection heuristics to auto-select LE vs BE when unspecified

## Implementation Checklist

- [ ] Extend utf16.rs with ByteOrder enum (LE, BE, Auto)
- [ ] Implement UTF-16BE byte sequence scanning
- [ ] Create confidence scoring function with weighted heuristics
- [ ] Add null-pattern detection and scoring penalties
- [ ] Implement configurable confidence threshold
- [ ] Add unit tests for UTF-16BE extraction
- [ ] Add unit tests for confidence scoring edge cases
- [ ] Add tests for false positive prevention (binary tables, numeric data)
- [ ] Add integration test with real-world UTF-16BE samples
- [ ] Document confidence scoring algorithm in code comments

## Test Cases

1. **Legitimate UTF-16BE:** Java .class constant pool strings
2. **False Positive:** Null-interleaved numeric table data
3. **Edge Case:** Mixed ASCII and Unicode characters
4. **Boundary:** Minimum length UTF-16BE strings at section edges
5. **Performance:** Large binary with scattered null bytes

## Dependencies

- **Blocked by:** #11 (UTF-16LE String Extraction) - Need base UTF-16 framework first
- **Related to:** #22 (Encoding Confidence Scoring and Ranking System)

## Success Criteria

- Correctly extracts UTF-16BE strings from Java .class files
- Rejects false positives from binary data with <10% false positive rate
- Confidence scores correlate with human assessment (validated against test corpus)
- Performance impact: <20% overhead compared to basic UTF-16 scanning
- Integration with ranking system properly weighs UTF-16BE confidence

## Task ID

stringy-analyzer/utf16be-support-confidence-scoring"

Length of output: 106

🏁 Script executed:

#!/bin/bash
# Add priority:medium label since it's part of the extraction pipeline
gh issue edit 12 --repo EvilBit-Labs/StringyMcStringFace --add-label "priority:medium"

Length of output: 61

🏁 Script executed:

#!/bin/bash
# Verify the updated issue
gh issue view 12 --repo EvilBit-Labs/StringyMcStringFace --json title,body,labels,milestone,assignees

Length of output: 6261

@unclesp1d3r I've successfully enhanced issue #12 with comprehensive details. Here's what was improved:

✅ Enhanced Title

Before: "UTF-16BE Support and Confidence Scoring"
After: "Implement UTF-16BE String Extraction with Confidence Scoring and False Positive Prevention"

✅ Enhanced Content

The issue body now includes:

Background Context

Explained UTF-16BE use cases (Java .class files, network protocols, big-endian architectures)
Clarified why confidence scoring is necessary to prevent false positives

Problem Statement

Detailed false positive scenarios (binary tables, numeric data, padding patterns, compressed data)
Explained current gaps in the extraction framework

Proposed Solution

Architecture: Unified UTF-16 extractor structure for both LE and BE with code examples
Confidence Scoring Algorithm: Detailed heuristics with positive/negative indicators and scoring formula
Null-Interleaved Pattern Detection: Specific checks for common false-positive patterns
Integration Points: CLI arguments, pipeline integration, section boundary handling

Implementation Details

Complete implementation checklist with 10 concrete tasks
5 specific test cases covering legitimate strings, false positives, and edge cases
Clear dependency chain (blocked by Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering #11, related to Implement Encoding Confidence Scoring and Integrate Section Weights into Ranking System #22)
Measurable success criteria (<10% false positive rate, <20% performance overhead)

✅ Updated Metadata

Labels Added: priority:medium (joins existing area:analyzer, type:enhancement, status:backlog, lang:rust, needs:tests)
Preserved: Milestone v0.1, assigned to you

The issue is now ready for implementation with clear technical guidance and acceptance criteria. You can view the updated issue here: #12

@traycerai generate branch:implement-utf-16le-string-extraction-with-confidence-scoring-and-minimum-length-filtering

@unclesp1d3r

Hi @unclesp1d3r, I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The codebase has a well-established extraction framework with ASCII extraction fully implemented, including confidence scoring via noise filters. UTF-16LE decoding exists in pe_resources.rs but only for resource extraction, not general section scanning. The Encoding enum already includes Utf16Le and Utf16Be variants. The ExtractionConfig has min_wide_length (default 3) ready for UTF-16 use. The BasicExtractor orchestrates extraction by calling format-specific extractors and applying noise filtering with configurable confidence thresholds.

Approach

Create a new utf16.rs module following the ASCII extractor pattern. Implement byte-order-aware scanning for both UTF-16LE and UTF-16BE with a ByteOrder enum (LE, BE, Auto). Add UTF-16-specific confidence scoring to detect false positives from null-interleaved binary data, then integrate with existing noise filters. Wire into BasicExtractor::extract_from_section to enable UTF-16 extraction alongside ASCII/UTF-8. Add comprehensive tests covering legitimate UTF-16 strings, false positive scenarios, and edge cases.

Reasoning

Listed the repository structure to understand the project layout. Read the extraction module files to understand the current architecture and patterns. Examined the ASCII extractor implementation to understand the extraction and confidence scoring approach. Reviewed the noise filtering framework to understand how confidence scores are calculated and combined. Checked the PE resources module to see existing UTF-16LE decoding logic. Searched for UTF-16 references to understand current usage. Reviewed integration tests to understand testing patterns and expectations.

Proposed File Changes
📄 src/extraction/utf16.rs ^(NEW) 🔗
References

📄 src/extraction/ascii.rs

📄 src/extraction/config.rs

📄 src/extraction/filters.rs

📄 src/types.rs

Create a new UTF-16 string extraction module following the pattern established in src/extraction/ascii.rs.

Module Structure:
ByteOrder enum: Define an enum with variants LE (little-endian), BE (big-endian), and Auto (detect both). This controls which byte orders to scan for.

Utf16ExtractionConfig struct: Create a configuration struct with fields:

min_length: usize - minimum string length in UTF-16 code units (default 3)

max_length: Option<usize> - maximum string length in UTF-16 code units

byte_order: ByteOrder - which byte order(s) to scan (default Auto)

confidence_threshold: f32 - minimum UTF-16-specific confidence (default 0.5)

Implement Default trait with sensible defaults

Add constructor new(min_length: usize) similar to ASCII config
UTF-16 confidence scoring functions:

calculate_utf16_confidence(chars: &[u16], byte_order: ByteOrder) -> f32 - main confidence calculator

check_valid_unicode_range(chars: &[u16]) -> f32 - validate code points are in valid Unicode ranges, penalize private use areas and invalid surrogates

check_null_pattern(chars: &[u16]) -> f32 - detect suspicious null patterns (every-other-null, fixed-offset nulls, excessive nulls)

check_ascii_ratio(chars: &[u16]) -> f32 - calculate ratio of ASCII-range characters (U+0020-U+007E), boost confidence for ASCII-heavy strings

check_printable_ratio(chars: &[u16]) -> f32 - ratio of printable characters including common Unicode ranges

check_byte_order_consistency(data: &[u8], byte_order: ByteOrder) -> f32 - verify byte order is consistent throughout

Combine these heuristics using weighted formula:
confidence = (valid_unicode_weight × valid_ratio)
           + (printable_weight × printable_ratio) 
           + (ascii_weight × ascii_ratio)
           - (null_pattern_penalty)
           - (invalid_range_penalty)
Clamp result to 0.0-1.0 range.
Core extraction functions:

extract_utf16_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - main extraction function that calls both LE and BE extractors based on config.byte_order

extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - scan for UTF-16LE sequences

extract_utf16be_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString> - scan for UTF-16BE sequences

extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString> - section-aware extraction with metadata population, similar to ascii::extract_from_section

Helper functions:

decode_utf16le(bytes: &[u8]) -> Result<(String, Vec<u16>)> - decode UTF-16LE bytes to String and return both the string and u16 vector for confidence scoring

decode_utf16be(bytes: &[u8]) -> Result<(String, Vec<u16>)> - decode UTF-16BE bytes to String and return both

is_valid_utf16_sequence(chars: &[u16]) -> bool - validate UTF-16 surrogate pairs and code points
Scanning Algorithm:
For each byte order:

Iterate through data in 2-byte chunks

Accumulate u16 values while they form valid UTF-16 sequences

When encountering invalid sequence or null terminator, check accumulated length

If length >= min_length, calculate UTF-16-specific confidence

If confidence >= threshold, decode to UTF-8 string

Create FoundString with appropriate encoding (Utf16Le or Utf16Be)

Handle end-of-buffer edge case

Confidence Scoring Details:

Valid Unicode ranges: U+0020-U+D7FF, U+E000-U+FFFD (exclude surrogates U+D800-U+DFFF, control chars U+0000-U+001F)

Null pattern detection: flag if >30% nulls, or nulls at regular intervals (every 2nd, 4th, 8th position)

ASCII ratio: boost confidence if >50% characters are in ASCII printable range

Byte order consistency: for Auto mode, ensure the detected byte order is consistent throughout the string

Integration with existing filters:
After UTF-16-specific confidence scoring, the decoded UTF-8 string should be passed through the existing CompositeNoiseFilter for additional validation. The final confidence should be the minimum of UTF-16 confidence and noise filter confidence, or a weighted combination.

Add comprehensive unit tests covering:

Basic UTF-16LE extraction

Basic UTF-16BE extraction

Auto byte order detection

Confidence scoring for legitimate strings

False positive detection (null-interleaved binary data)

Edge cases (minimum length, odd-length data, invalid surrogates)

Section-aware extraction with metadata
📄 src/extraction/mod.rs ^(MODIFY) 🔗
References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

Integrate the new UTF-16 extraction module into the extraction framework.

Changes needed:
Module declaration (around line 99): Add pub mod utf16; after the existing module declarations.
Re-exports (around line 105): Add public re-exports for the UTF-16 module:
pub use utf16::{ByteOrder, Utf16ExtractionConfig, extract_utf16_strings, extract_from_section as extract_utf16_from_section};
ExtractionConfig struct (around lines 133-160): Add new fields for UTF-16 configuration:

utf16_byte_order: ByteOrder - which byte order(s) to scan (default Auto)

utf16_confidence_threshold: f32 - minimum UTF-16-specific confidence (default 0.5)

Update the Default impl to initialize these fields with sensible defaults.

ExtractionConfig::validate (around lines 185-212): Add validation for the new UTF-16 fields:

Ensure utf16_confidence_threshold is between 0.0 and 1.0

BasicExtractor::extract_from_section (around lines 416-537): Add UTF-16 extraction logic after the UTF-8 extraction block (around line 534):

Check if UTF-16LE or UTF-16BE is enabled in config.encodings or config.enabled_encodings

Create Utf16ExtractionConfig from the main config

Call utf16::extract_from_section with appropriate parameters

Merge results into found_strings vector

Apply the same noise filtering and confidence threshold logic as ASCII/UTF-8
Integration approach:
Follow the same pattern used for ASCII extraction:

Build UTF-16 config from extraction config

Call the UTF-16 extractor with section info

The UTF-16 extractor handles its own confidence scoring

Results are returned as FoundString with proper metadata

Optionally apply additional noise filtering if enabled

Encoding filtering:
Check both config.encodings and config.enabled_encodings for Encoding::Utf16Le and Encoding::Utf16Be to determine which byte orders to scan. If ByteOrder::Auto is configured, scan for both but only if at least one is enabled in the config.
📄 tests/test_utf16_extraction.rs ^(NEW) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

📄 tests/integration_extraction.rs ^(MODIFY)

Create comprehensive unit tests for UTF-16 extraction functionality.

Test categories:

Basic extraction tests:

test_extract_utf16le_basic - extract simple UTF-16LE strings ("Hello", "World")

test_extract_utf16be_basic - extract simple UTF-16BE strings

test_extract_utf16_auto_detects_le - Auto mode correctly detects LE

test_extract_utf16_auto_detects_be - Auto mode correctly detects BE

test_extract_utf16_mixed_ascii_unicode - strings with both ASCII and Unicode characters (e.g., "Hello 世界")

Minimum length filtering:

test_utf16_min_length_filtering - verify strings below min_length are filtered

test_utf16_exact_min_length - strings exactly at min_length are included

test_utf16_custom_min_length - test with custom min_length values

Confidence scoring tests:

test_utf16_confidence_legitimate_string - high confidence for normal text

test_utf16_confidence_ascii_heavy - boost for ASCII-heavy strings

test_utf16_confidence_null_pattern_penalty - low confidence for null-interleaved data

test_utf16_confidence_invalid_unicode - low confidence for invalid code points

test_utf16_confidence_threshold_filtering - strings below threshold are filtered

False positive prevention:

test_utf16_false_positive_binary_table - reject null-interleaved numeric table data

test_utf16_false_positive_every_other_null - reject patterns like "A\0B\0C\0" that could be ASCII or coincidental

test_utf16_false_positive_padding - reject padding bytes that look like UTF-16

test_utf16_false_positive_fixed_offset_nulls - reject structured binary data with nulls at regular intervals

Edge cases:

test_utf16_odd_length_data - handle odd-length byte arrays gracefully

test_utf16_invalid_surrogates - handle invalid surrogate pairs

test_utf16_string_at_buffer_start - string starting at offset 0

test_utf16_string_at_buffer_end - string ending at buffer boundary

test_utf16_empty_data - empty input returns empty result

test_utf16_max_length_filtering - strings exceeding max_length are filtered

Section-aware extraction:

test_utf16_extract_from_section_metadata - verify section name, offset, RVA are populated correctly

test_utf16_extract_from_section_offset_adjustment - absolute offsets calculated correctly

test_utf16_extract_from_section_rva_calculation - RVA calculated from section base + relative offset

test_utf16_extract_from_section_bounds_checking - handle section boundaries correctly

Real-world scenarios:

test_utf16_java_class_constant_pool - extract strings from Java .class file format (UTF-16BE)

test_utf16_windows_pe_strings - extract UTF-16LE strings from PE binary sections

test_utf16_mixed_encodings - data with both ASCII and UTF-16 strings

Byte order tests:

test_byte_order_le_only - ByteOrder::LE only scans little-endian

test_byte_order_be_only - ByteOrder::BE only scans big-endian

test_byte_order_auto_finds_both - ByteOrder::Auto finds both LE and BE strings

Test data examples:

UTF-16LE "Hello": [0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F, 0x00]

UTF-16BE "Hello": [0x00, 0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F]

False positive (null-interleaved): [0x41, 0x00, 0x42, 0x00, 0x43, 0x00] (could be "ABC" in ASCII or UTF-16LE)

Binary table: [0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04, 0x00] (numeric data)

Use the same testing patterns as src/extraction/ascii.rs tests for consistency.
📄 tests/integration_extraction.rs ^(MODIFY) 🔗
References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Add integration tests for UTF-16 extraction within the existing extraction test suite.

New test functions to add:

test_basic_extractor_utf16le_strings - Test UTF-16LE extraction through BasicExtractor:

Create test data with embedded UTF-16LE strings

Create a SectionInfo with appropriate metadata

Configure ExtractionConfig to enable UTF-16LE encoding

Call extractor.extract_from_section

Verify extracted strings have correct encoding, text, offsets, and metadata

test_basic_extractor_utf16be_strings - Test UTF-16BE extraction:

Similar to UTF-16LE test but with big-endian byte order

Verify Encoding::Utf16Be is set correctly

test_basic_extractor_utf16_auto_detection - Test automatic byte order detection:

Create data with both UTF-16LE and UTF-16BE strings

Configure with ByteOrder::Auto

Verify both are extracted with correct encodings

test_basic_extractor_utf16_encoding_filtering - Test encoding selection:

Configure to only allow UTF-16LE (exclude UTF-16BE)

Verify only UTF-16LE strings are extracted

Test the reverse (only UTF-16BE)

test_basic_extractor_utf16_confidence_filtering - Test confidence threshold:

Create data with legitimate UTF-16 strings and false positives

Set a confidence threshold (e.g., 0.6)

Verify legitimate strings pass and false positives are filtered

test_basic_extractor_utf16_with_noise_filtering - Test integration with noise filters:

Enable noise filtering in config

Create UTF-16 strings with varying characteristics (normal text, repetitive, high entropy)

Verify noise filtering affects final confidence scores

test_basic_extractor_mixed_encodings - Test extraction of multiple encodings:

Create data with ASCII, UTF-8, UTF-16LE, and UTF-16BE strings

Enable all encodings in config

Verify all are extracted with correct encodings

Integration with existing tests:
Add these tests after the existing UTF-8 tests (around line 80) to maintain logical grouping. Follow the same patterns:

Use helper function get_fixture_path if testing with real binaries

Create SectionInfo with appropriate section types and weights

Use ExtractionConfig with explicit encoding configuration

Verify all FoundString fields are properly populated

Check confidence scores are reasonable

Configuration example:
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    enabled_encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    min_wide_length: 3,
    utf16_byte_order: ByteOrder::Auto,
    utf16_confidence_threshold: 0.5,
    noise_filtering_enabled: true,
    ..Default::default()
};
📄 src/extraction/pe_resources.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

Refactor the existing decode_utf16le function to be reusable and add UTF-16BE support.

Changes needed:

Move decode_utf16le to utf16.rs (optional but recommended): The decode_utf16le function (lines 372-390) could be moved to the new utf16.rs module and made public, then re-imported here. This promotes code reuse and keeps UTF-16 decoding logic centralized. If moved, update the function signature to return Result<String> and handle the null-trimming consistently.

Alternative - Keep as-is: If not moving the function, ensure the new utf16.rs module implements its own decoding functions that are consistent with this implementation. The key logic to maintain:

Handle odd-length input by truncating

Use u16::from_le_bytes for LE, u16::from_be_bytes for BE

Use String::from_utf16 for decoding

Trim trailing null terminators

Return appropriate errors for invalid UTF-16

Update decode_manifest_data function (lines 657-696): The UTF-16BE decoding logic (lines 687-696) is already implemented here. This can serve as a reference for the utf16.rs implementation. Ensure consistency in error handling and null-trimming behavior.

Documentation updates: Add a note in the module documentation (around line 1) mentioning that general UTF-16 extraction is now handled by the utf16 module, while this module focuses on PE-specific resource extraction.

No breaking changes: The existing functions should continue to work as-is. The new utf16.rs module will provide general-purpose UTF-16 extraction, while this module remains focused on PE resource-specific extraction.
📄 docs/src/string-extraction.md ^(MODIFY) 🔗
References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update the string extraction documentation to include UTF-16 support.

Sections to update:
Supported Encodings section: Add UTF-16LE and UTF-16BE to the list of supported encodings:

UTF-16LE (Little-Endian): Common in Windows PE binaries, default 3 character minimum

UTF-16BE (Big-Endian): Found in Java .class files, network protocols, some cross-platform binaries

Automatic byte order detection available

Confidence Scoring section: Add a subsection on UTF-16-specific confidence scoring:

Explain the false positive problem with null-interleaved binary data

Describe the heuristics used: valid Unicode ranges, null pattern detection, ASCII ratio, printable character ratio

Mention the configurable confidence threshold

Explain how UTF-16 confidence combines with general noise filtering

Configuration section: Document the new UTF-16 configuration options:

min_wide_length - minimum length for UTF-16 strings (default 3)

utf16_byte_order - which byte order(s) to scan (LE, BE, Auto)

utf16_confidence_threshold - minimum UTF-16-specific confidence (default 0.5)

encodings - include Utf16Le and/or Utf16Be to enable UTF-16 extraction
Examples section: Add examples of UTF-16 extraction:
// Extract UTF-16LE strings from Windows PE binary
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le],
    min_wide_length: 3,
    utf16_confidence_threshold: 0.6,
    ..Default::default()
};

// Extract both UTF-16LE and UTF-16BE with auto-detection
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    utf16_byte_order: ByteOrder::Auto,
    ..Default::default()
};
False Positive Prevention section: Add a new section explaining:

Why UTF-16 extraction is prone to false positives

Examples of binary data that can look like UTF-16 (null-interleaved tables, padding)

How the confidence scoring system mitigates this

Recommendations for tuning the confidence threshold

Performance Considerations section: Mention that UTF-16 scanning adds overhead:

Scanning for both byte orders doubles the work

Confidence scoring adds computational cost

Recommend using specific byte order (LE or BE) when the target format is known

Auto mode is best for unknown or mixed-format binaries
Diagram
sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module
    participant ConfidenceScorer
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if UTF-16 enabled in config
    
    alt UTF-16LE or UTF-16BE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, utf16_config)
        
        alt ByteOrder::Auto
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        else ByteOrder::LE
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
        else ByteOrder::BE
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        end
        
        loop For each candidate string
            Utf16Module->>ConfidenceScorer: calculate_utf16_confidence(chars, byte_order)
            ConfidenceScorer->>ConfidenceScorer: check_valid_unicode_range()
            ConfidenceScorer->>ConfidenceScorer: check_null_pattern()
            ConfidenceScorer->>ConfidenceScorer: check_ascii_ratio()
            ConfidenceScorer->>ConfidenceScorer: check_printable_ratio()
            ConfidenceScorer-->>Utf16Module: utf16_confidence (0.0-1.0)
            
            alt utf16_confidence >= threshold
                Utf16Module->>Utf16Module: decode_to_utf8(chars)
                
                opt Noise filtering enabled
                    Utf16Module->>NoiseFilter: calculate_confidence(text, context)
                    NoiseFilter-->>Utf16Module: noise_confidence (0.0-1.0)
                    Utf16Module->>Utf16Module: final_confidence = min(utf16_conf, noise_conf)
                end
                
                alt final_confidence >= min_threshold
                    Utf16Module->>FoundString: Create with metadata
                    FoundString-->>Utf16Module: found_string
                end
            end
        end
        
        Utf16Module-->>BasicExtractor: Vec<FoundString>
        BasicExtractor->>BasicExtractor: Merge with ASCII/UTF-8 results
    end
    
    BasicExtractor-->>User: Vec<FoundString>
Loading
Import In IDE
🤖 Prompt for AI Agents
I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established extraction framework with ASCII extraction fully implemented, including confidence scoring via noise filters. UTF-16LE decoding exists in `pe_resources.rs` but only for resource extraction, not general section scanning. The `Encoding` enum already includes `Utf16Le` and `Utf16Be` variants. The `ExtractionConfig` has `min_wide_length` (default 3) ready for UTF-16 use. The `BasicExtractor` orchestrates extraction by calling format-specific extractors and applying noise filtering with configurable confidence thresholds.

### Approach

Create a new `utf16.rs` module following the ASCII extractor pattern. Implement byte-order-aware scanning for both UTF-16LE and UTF-16BE with a `ByteOrder` enum (LE, BE, Auto). Add UTF-16-specific confidence scoring to detect false positives from null-interleaved binary data, then integrate with existing noise filters. Wire into `BasicExtractor::extract_from_section` to enable UTF-16 extraction alongside ASCII/UTF-8. Add comprehensive tests covering legitimate UTF-16 strings, false positive scenarios, and edge cases.

### Reasoning

Listed the repository structure to understand the project layout. Read the extraction module files to understand the current architecture and patterns. Examined the ASCII extractor implementation to understand the extraction and confidence scoring approach. Reviewed the noise filtering framework to understand how confidence scores are calculated and combined. Checked the PE resources module to see existing UTF-16LE decoding logic. Searched for UTF-16 references to understand current usage. Reviewed integration tests to understand testing patterns and expectations.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module
    participant ConfidenceScorer
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if UTF-16 enabled in config
    
    alt UTF-16LE or UTF-16BE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, utf16_config)
        
        alt ByteOrder::Auto
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        else ByteOrder::LE
            Utf16Module->>Utf16Module: extract_utf16le_strings(data)
        else ByteOrder::BE
            Utf16Module->>Utf16Module: extract_utf16be_strings(data)
        end
        
        loop For each candidate string
            Utf16Module->>ConfidenceScorer: calculate_utf16_confidence(chars, byte_order)
            ConfidenceScorer->>ConfidenceScorer: check_valid_unicode_range()
            ConfidenceScorer->>ConfidenceScorer: check_null_pattern()
            ConfidenceScorer->>ConfidenceScorer: check_ascii_ratio()
            ConfidenceScorer->>ConfidenceScorer: check_printable_ratio()
            ConfidenceScorer-->>Utf16Module: utf16_confidence (0.0-1.0)
            
            alt utf16_confidence >= threshold
                Utf16Module->>Utf16Module: decode_to_utf8(chars)
                
                opt Noise filtering enabled
                    Utf16Module->>NoiseFilter: calculate_confidence(text, context)
                    NoiseFilter-->>Utf16Module: noise_confidence (0.0-1.0)
                    Utf16Module->>Utf16Module: final_confidence = min(utf16_conf, noise_conf)
                end
                
                alt final_confidence >= min_threshold
                    Utf16Module->>FoundString: Create with metadata
                    FoundString-->>Utf16Module: found_string
                end
            end
        end
        
        Utf16Module-->>BasicExtractor: Vec<FoundString>
        BasicExtractor->>BasicExtractor: Merge with ASCII/UTF-8 results
    end
    
    BasicExtractor-->>User: Vec<FoundString>

## Proposed File Changes

### src/extraction/utf16.rs(NEW)

References: 

- src/extraction/ascii.rs
- src/extraction/config.rs
- src/extraction/filters.rs
- src/types.rs

Create a new UTF-16 string extraction module following the pattern established in `src/extraction/ascii.rs`.

**Module Structure:**

1. **ByteOrder enum:** Define an enum with variants `LE` (little-endian), `BE` (big-endian), and `Auto` (detect both). This controls which byte orders to scan for.

2. **Utf16ExtractionConfig struct:** Create a configuration struct with fields:
   - `min_length: usize` - minimum string length in UTF-16 code units (default 3)
   - `max_length: Option<usize>` - maximum string length in UTF-16 code units
   - `byte_order: ByteOrder` - which byte order(s) to scan (default Auto)
   - `confidence_threshold: f32` - minimum UTF-16-specific confidence (default 0.5)
   - Implement `Default` trait with sensible defaults
   - Add constructor `new(min_length: usize)` similar to ASCII config

3. **UTF-16 confidence scoring functions:**
   - `calculate_utf16_confidence(chars: &[u16], byte_order: ByteOrder) -> f32` - main confidence calculator
   - `check_valid_unicode_range(chars: &[u16]) -> f32` - validate code points are in valid Unicode ranges, penalize private use areas and invalid surrogates
   - `check_null_pattern(chars: &[u16]) -> f32` - detect suspicious null patterns (every-other-null, fixed-offset nulls, excessive nulls)
   - `check_ascii_ratio(chars: &[u16]) -> f32` - calculate ratio of ASCII-range characters (U+0020-U+007E), boost confidence for ASCII-heavy strings
   - `check_printable_ratio(chars: &[u16]) -> f32` - ratio of printable characters including common Unicode ranges
   - `check_byte_order_consistency(data: &[u8], byte_order: ByteOrder) -> f32` - verify byte order is consistent throughout
   
   Combine these heuristics using weighted formula:
   ```
   confidence = (valid_unicode_weight × valid_ratio)
              + (printable_weight × printable_ratio) 
              + (ascii_weight × ascii_ratio)
              - (null_pattern_penalty)
              - (invalid_range_penalty)
   ```
   Clamp result to 0.0-1.0 range.

4. **Core extraction functions:**
   - `extract_utf16_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - main extraction function that calls both LE and BE extractors based on config.byte_order
   - `extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - scan for UTF-16LE sequences
   - `extract_utf16be_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>` - scan for UTF-16BE sequences
   - `extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>` - section-aware extraction with metadata population, similar to `ascii::extract_from_section`

5. **Helper functions:**
   - `decode_utf16le(bytes: &[u8]) -> Result<(String, Vec<u16>)>` - decode UTF-16LE bytes to String and return both the string and u16 vector for confidence scoring
   - `decode_utf16be(bytes: &[u8]) -> Result<(String, Vec<u16>)>` - decode UTF-16BE bytes to String and return both
   - `is_valid_utf16_sequence(chars: &[u16]) -> bool` - validate UTF-16 surrogate pairs and code points

**Scanning Algorithm:**
For each byte order:
1. Iterate through data in 2-byte chunks
2. Accumulate u16 values while they form valid UTF-16 sequences
3. When encountering invalid sequence or null terminator, check accumulated length
4. If length >= min_length, calculate UTF-16-specific confidence
5. If confidence >= threshold, decode to UTF-8 string
6. Create FoundString with appropriate encoding (Utf16Le or Utf16Be)
7. Handle end-of-buffer edge case

**Confidence Scoring Details:**
- Valid Unicode ranges: U+0020-U+D7FF, U+E000-U+FFFD (exclude surrogates U+D800-U+DFFF, control chars U+0000-U+001F)
- Null pattern detection: flag if >30% nulls, or nulls at regular intervals (every 2nd, 4th, 8th position)
- ASCII ratio: boost confidence if >50% characters are in ASCII printable range
- Byte order consistency: for Auto mode, ensure the detected byte order is consistent throughout the string

**Integration with existing filters:**
After UTF-16-specific confidence scoring, the decoded UTF-8 string should be passed through the existing `CompositeNoiseFilter` for additional validation. The final confidence should be the minimum of UTF-16 confidence and noise filter confidence, or a weighted combination.

Add comprehensive unit tests covering:
- Basic UTF-16LE extraction
- Basic UTF-16BE extraction
- Auto byte order detection
- Confidence scoring for legitimate strings
- False positive detection (null-interleaved binary data)
- Edge cases (minimum length, odd-length data, invalid surrogates)
- Section-aware extraction with metadata

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs

Integrate the new UTF-16 extraction module into the extraction framework.

**Changes needed:**

1. **Module declaration (around line 99):** Add `pub mod utf16;` after the existing module declarations.

2. **Re-exports (around line 105):** Add public re-exports for the UTF-16 module:
   ```rust
   pub use utf16::{ByteOrder, Utf16ExtractionConfig, extract_utf16_strings, extract_from_section as extract_utf16_from_section};
   ```

3. **ExtractionConfig struct (around lines 133-160):** Add new fields for UTF-16 configuration:
   - `utf16_byte_order: ByteOrder` - which byte order(s) to scan (default Auto)
   - `utf16_confidence_threshold: f32` - minimum UTF-16-specific confidence (default 0.5)
   
   Update the `Default` impl to initialize these fields with sensible defaults.

4. **ExtractionConfig::validate (around lines 185-212):** Add validation for the new UTF-16 fields:
   - Ensure `utf16_confidence_threshold` is between 0.0 and 1.0

5. **BasicExtractor::extract_from_section (around lines 416-537):** Add UTF-16 extraction logic after the UTF-8 extraction block (around line 534):
   - Check if UTF-16LE or UTF-16BE is enabled in `config.encodings` or `config.enabled_encodings`
   - Create `Utf16ExtractionConfig` from the main config
   - Call `utf16::extract_from_section` with appropriate parameters
   - Merge results into `found_strings` vector
   - Apply the same noise filtering and confidence threshold logic as ASCII/UTF-8

**Integration approach:**
Follow the same pattern used for ASCII extraction:
1. Build UTF-16 config from extraction config
2. Call the UTF-16 extractor with section info
3. The UTF-16 extractor handles its own confidence scoring
4. Results are returned as `FoundString` with proper metadata
5. Optionally apply additional noise filtering if enabled

**Encoding filtering:**
Check both `config.encodings` and `config.enabled_encodings` for `Encoding::Utf16Le` and `Encoding::Utf16Be` to determine which byte orders to scan. If `ByteOrder::Auto` is configured, scan for both but only if at least one is enabled in the config.

### tests/test_utf16_extraction.rs(NEW)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs
- tests/integration_extraction.rs(MODIFY)

Create comprehensive unit tests for UTF-16 extraction functionality.

**Test categories:**

1. **Basic extraction tests:**
   - `test_extract_utf16le_basic` - extract simple UTF-16LE strings ("Hello", "World")
   - `test_extract_utf16be_basic` - extract simple UTF-16BE strings
   - `test_extract_utf16_auto_detects_le` - Auto mode correctly detects LE
   - `test_extract_utf16_auto_detects_be` - Auto mode correctly detects BE
   - `test_extract_utf16_mixed_ascii_unicode` - strings with both ASCII and Unicode characters (e.g., "Hello 世界")

2. **Minimum length filtering:**
   - `test_utf16_min_length_filtering` - verify strings below min_length are filtered
   - `test_utf16_exact_min_length` - strings exactly at min_length are included
   - `test_utf16_custom_min_length` - test with custom min_length values

3. **Confidence scoring tests:**
   - `test_utf16_confidence_legitimate_string` - high confidence for normal text
   - `test_utf16_confidence_ascii_heavy` - boost for ASCII-heavy strings
   - `test_utf16_confidence_null_pattern_penalty` - low confidence for null-interleaved data
   - `test_utf16_confidence_invalid_unicode` - low confidence for invalid code points
   - `test_utf16_confidence_threshold_filtering` - strings below threshold are filtered

4. **False positive prevention:**
   - `test_utf16_false_positive_binary_table` - reject null-interleaved numeric table data
   - `test_utf16_false_positive_every_other_null` - reject patterns like "A\0B\0C\0" that could be ASCII or coincidental
   - `test_utf16_false_positive_padding` - reject padding bytes that look like UTF-16
   - `test_utf16_false_positive_fixed_offset_nulls` - reject structured binary data with nulls at regular intervals

5. **Edge cases:**
   - `test_utf16_odd_length_data` - handle odd-length byte arrays gracefully
   - `test_utf16_invalid_surrogates` - handle invalid surrogate pairs
   - `test_utf16_string_at_buffer_start` - string starting at offset 0
   - `test_utf16_string_at_buffer_end` - string ending at buffer boundary
   - `test_utf16_empty_data` - empty input returns empty result
   - `test_utf16_max_length_filtering` - strings exceeding max_length are filtered

6. **Section-aware extraction:**
   - `test_utf16_extract_from_section_metadata` - verify section name, offset, RVA are populated correctly
   - `test_utf16_extract_from_section_offset_adjustment` - absolute offsets calculated correctly
   - `test_utf16_extract_from_section_rva_calculation` - RVA calculated from section base + relative offset
   - `test_utf16_extract_from_section_bounds_checking` - handle section boundaries correctly

7. **Real-world scenarios:**
   - `test_utf16_java_class_constant_pool` - extract strings from Java .class file format (UTF-16BE)
   - `test_utf16_windows_pe_strings` - extract UTF-16LE strings from PE binary sections
   - `test_utf16_mixed_encodings` - data with both ASCII and UTF-16 strings

8. **Byte order tests:**
   - `test_byte_order_le_only` - ByteOrder::LE only scans little-endian
   - `test_byte_order_be_only` - ByteOrder::BE only scans big-endian  
   - `test_byte_order_auto_finds_both` - ByteOrder::Auto finds both LE and BE strings

**Test data examples:**
- UTF-16LE "Hello": `[0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F, 0x00]`
- UTF-16BE "Hello": `[0x00, 0x48, 0x00, 0x65, 0x00, 0x6C, 0x00, 0x6C, 0x00, 0x6F]`
- False positive (null-interleaved): `[0x41, 0x00, 0x42, 0x00, 0x43, 0x00]` (could be "ABC" in ASCII or UTF-16LE)
- Binary table: `[0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04, 0x00]` (numeric data)

Use the same testing patterns as `src/extraction/ascii.rs` tests for consistency.

### tests/integration_extraction.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Add integration tests for UTF-16 extraction within the existing extraction test suite.

**New test functions to add:**

1. **test_basic_extractor_utf16le_strings** - Test UTF-16LE extraction through BasicExtractor:
   - Create test data with embedded UTF-16LE strings
   - Create a SectionInfo with appropriate metadata
   - Configure ExtractionConfig to enable UTF-16LE encoding
   - Call `extractor.extract_from_section`
   - Verify extracted strings have correct encoding, text, offsets, and metadata

2. **test_basic_extractor_utf16be_strings** - Test UTF-16BE extraction:
   - Similar to UTF-16LE test but with big-endian byte order
   - Verify `Encoding::Utf16Be` is set correctly

3. **test_basic_extractor_utf16_auto_detection** - Test automatic byte order detection:
   - Create data with both UTF-16LE and UTF-16BE strings
   - Configure with `ByteOrder::Auto`
   - Verify both are extracted with correct encodings

4. **test_basic_extractor_utf16_encoding_filtering** - Test encoding selection:
   - Configure to only allow UTF-16LE (exclude UTF-16BE)
   - Verify only UTF-16LE strings are extracted
   - Test the reverse (only UTF-16BE)

5. **test_basic_extractor_utf16_confidence_filtering** - Test confidence threshold:
   - Create data with legitimate UTF-16 strings and false positives
   - Set a confidence threshold (e.g., 0.6)
   - Verify legitimate strings pass and false positives are filtered

6. **test_basic_extractor_utf16_with_noise_filtering** - Test integration with noise filters:
   - Enable noise filtering in config
   - Create UTF-16 strings with varying characteristics (normal text, repetitive, high entropy)
   - Verify noise filtering affects final confidence scores

7. **test_basic_extractor_mixed_encodings** - Test extraction of multiple encodings:
   - Create data with ASCII, UTF-8, UTF-16LE, and UTF-16BE strings
   - Enable all encodings in config
   - Verify all are extracted with correct encodings

**Integration with existing tests:**
Add these tests after the existing UTF-8 tests (around line 80) to maintain logical grouping. Follow the same patterns:
- Use helper function `get_fixture_path` if testing with real binaries
- Create `SectionInfo` with appropriate section types and weights
- Use `ExtractionConfig` with explicit encoding configuration
- Verify all `FoundString` fields are properly populated
- Check confidence scores are reasonable

**Configuration example:**
```rust
let config = ExtractionConfig {
    encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    enabled_encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
    min_wide_length: 3,
    utf16_byte_order: ByteOrder::Auto,
    utf16_confidence_threshold: 0.5,
    noise_filtering_enabled: true,
    ..Default::default()
};
```

### src/extraction/pe_resources.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)

Refactor the existing `decode_utf16le` function to be reusable and add UTF-16BE support.

**Changes needed:**

1. **Move decode_utf16le to utf16.rs (optional but recommended):** The `decode_utf16le` function (lines 372-390) could be moved to the new `utf16.rs` module and made public, then re-imported here. This promotes code reuse and keeps UTF-16 decoding logic centralized. If moved, update the function signature to return `Result<String>` and handle the null-trimming consistently.

2. **Alternative - Keep as-is:** If not moving the function, ensure the new `utf16.rs` module implements its own decoding functions that are consistent with this implementation. The key logic to maintain:
   - Handle odd-length input by truncating
   - Use `u16::from_le_bytes` for LE, `u16::from_be_bytes` for BE
   - Use `String::from_utf16` for decoding
   - Trim trailing null terminators
   - Return appropriate errors for invalid UTF-16

3. **Update decode_manifest_data function (lines 657-696):** The UTF-16BE decoding logic (lines 687-696) is already implemented here. This can serve as a reference for the `utf16.rs` implementation. Ensure consistency in error handling and null-trimming behavior.

4. **Documentation updates:** Add a note in the module documentation (around line 1) mentioning that general UTF-16 extraction is now handled by the `utf16` module, while this module focuses on PE-specific resource extraction.

**No breaking changes:** The existing functions should continue to work as-is. The new `utf16.rs` module will provide general-purpose UTF-16 extraction, while this module remains focused on PE resource-specific extraction.

### docs/src/string-extraction.md(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update the string extraction documentation to include UTF-16 support.

**Sections to update:**

1. **Supported Encodings section:** Add UTF-16LE and UTF-16BE to the list of supported encodings:
   - UTF-16LE (Little-Endian): Common in Windows PE binaries, default 3 character minimum
   - UTF-16BE (Big-Endian): Found in Java .class files, network protocols, some cross-platform binaries
   - Automatic byte order detection available

2. **Confidence Scoring section:** Add a subsection on UTF-16-specific confidence scoring:
   - Explain the false positive problem with null-interleaved binary data
   - Describe the heuristics used: valid Unicode ranges, null pattern detection, ASCII ratio, printable character ratio
   - Mention the configurable confidence threshold
   - Explain how UTF-16 confidence combines with general noise filtering

3. **Configuration section:** Document the new UTF-16 configuration options:
   - `min_wide_length` - minimum length for UTF-16 strings (default 3)
   - `utf16_byte_order` - which byte order(s) to scan (LE, BE, Auto)
   - `utf16_confidence_threshold` - minimum UTF-16-specific confidence (default 0.5)
   - `encodings` - include `Utf16Le` and/or `Utf16Be` to enable UTF-16 extraction

4. **Examples section:** Add examples of UTF-16 extraction:
   ```rust
   // Extract UTF-16LE strings from Windows PE binary
   let config = ExtractionConfig {
       encodings: vec![Encoding::Utf16Le],
       min_wide_length: 3,
       utf16_confidence_threshold: 0.6,
       ..Default::default()
   };
   
   // Extract both UTF-16LE and UTF-16BE with auto-detection
   let config = ExtractionConfig {
       encodings: vec![Encoding::Utf16Le, Encoding::Utf16Be],
       utf16_byte_order: ByteOrder::Auto,
       ..Default::default()
   };
   ```

5. **False Positive Prevention section:** Add a new section explaining:
   - Why UTF-16 extraction is prone to false positives
   - Examples of binary data that can look like UTF-16 (null-interleaved tables, padding)
   - How the confidence scoring system mitigates this
   - Recommendations for tuning the confidence threshold

6. **Performance Considerations section:** Mention that UTF-16 scanning adds overhead:
   - Scanning for both byte orders doubles the work
   - Confidence scoring adds computational cost
   - Recommend using specific byte order (LE or BE) when the target format is known
   - Auto mode is best for unknown or mixed-format binaries
Developer Humor

🎭 The UTF-16 Extractor's Tale\n>\n> In a world of bytes, both big and small,\n> Where endianness confounds us all,\n> A string extractor, brave and true,\n> Scans for patterns, old and new.\n>\n> \Is it LE or BE?\ it cries,\n> \Or just null bytes in disguise?\n> With confidence scores, it takes its stand,\n> Rejecting noise with a steady hand! 🔍✨

Execution Information

Branch: main
Commit: c50e9b6

Implement UTF-16BE String Extraction with Confidence Scoring and False Positive Prevention #12

Description

Summary

Background

Problem Statement

Requirements

Proposed Solution

1. Extend src/extraction/utf16.rs

2. Confidence Scoring Algorithm

3. Null-Interleaved Pattern Detection

4. Integration Points

Implementation Checklist

Test Cases

Dependencies

Success Criteria

Task ID

Activity

unclesp1d3r commented on Nov 10, 2025

traycerai commented on Nov 10, 2025

Plan

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

coderabbitai commented on Nov 10, 2025

✅ Enhanced Title

✅ Enhanced Content

Background Context

Problem Statement

Proposed Solution

Implementation Details

✅ Updated Metadata

unclesp1d3r commented on Nov 13, 2025

traycerai commented on Nov 13, 2025

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions