Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering

Summary

Implement robust UTF-16LE (Little Endian) string extraction capability to properly handle wide character strings commonly found in Windows PE binaries, .NET assemblies, and resource sections. This is a critical feature for analyzing Windows executables where UTF-16LE is the dominant string encoding.

Context

UTF-16LE strings are ubiquitous in Windows binaries:

PE Resources: VERSIONINFO, STRINGTABLE, RT_MANIFEST all use UTF-16LE
.NET Metadata: String literals in managed assemblies
Windows API Strings: Wide character (WCHAR) strings in native code
Registry Paths: Many internal registry references

The standard strings command often mangles UTF-16 strings or extracts them as ASCII with null bytes interleaved, making them difficult to read and classify. Proper UTF-16LE extraction will significantly improve Stringy's effectiveness on Windows binaries.

Technical Background

UTF-16LE Detection Heuristics

UTF-16LE strings can be detected using the following heuristics:

Even-length byte sequences (UTF-16 uses 2 bytes per character)
Low byte is printable ASCII (0x20-0x7E)
High byte is mostly zero (for basic multilingual plane characters)
Null termination pattern: Two consecutive zero bytes (0x00 0x00)

Confidence Scoring

To avoid false positives from coincidental byte patterns:

High confidence (>90%): >90% printable characters with proper null termination
Medium confidence (70-90%): >70% printable, reasonable length (>5 chars)
Low confidence (<70%): >50% printable, may be coincidental data

Requirements

Core Functionality

Create src/extraction/utf16.rs module with UTF-16LE extraction logic
Implement extract_utf16le_strings(data: &[u8], config: &ExtractionConfig) -> Vec<RawString>
Support configurable minimum character length (default: 3 characters = 6 bytes)
Implement confidence scoring algorithm
Handle proper null termination detection (0x00 0x00 sequence)
Return strings with accurate offset, length, and encoding metadata

Configuration

Add min_utf16_len: usize to ExtractionConfig (default: 3)
Add utf16_min_confidence: f32 threshold (default: 0.7)
Support section-aware minimum length overrides

Integration

Integrate UTF-16LE extraction into main extraction pipeline
Wire up UTF-16LE extractor in src/extraction/mod.rs
Ensure UTF-16LE strings are properly passed to classification module
Add UTF-16LE support to deduplication logic

Testing

Unit tests for basic UTF-16LE string extraction
Test cases for confidence scoring (high, medium, low)
Test handling of null termination
Test minimum length filtering
Test false positive rejection (random binary data)
Test with real-world Windows PE samples
Edge cases: partial strings at section boundaries, interleaved data

Proposed Implementation

File Structure

// src/extraction/utf16.rs

use crate::types::{Encoding, FoundString, StringSource};

/// Configuration for UTF-16LE extraction
pub struct Utf16Config {
    pub min_char_len: usize,
    pub min_confidence: f32,
}

/// Raw UTF-16LE string candidate with confidence score
struct Utf16Candidate {
    data: Vec<u8>,
    offset: usize,
    char_count: usize,
    confidence: f32,
}

/// Extract UTF-16LE strings from binary data
pub fn extract_utf16le_strings(
    data: &[u8],
    config: &Utf16Config,
    section_name: Option<String>,
) -> Vec<FoundString> {
    // Implementation based on algorithm in docs/src/string-extraction.md
    // 1. Scan for even-offset sequences with printable low bytes
    // 2. Calculate confidence scores
    // 3. Filter by minimum length and confidence
    // 4. Convert to FoundString with proper metadata
}

/// Calculate confidence score for a UTF-16LE candidate
fn calculate_confidence(data: &[u8]) -> f32 {
    // Percentage of valid printable UTF-16LE characters
    // Bonus for proper null termination
    // Penalty for excessive high bytes or invalid patterns
}

/// Check if a UTF-16LE character is printable
fn is_printable_utf16le_char(low: u8, high: u8) -> bool {
    // Basic multilingual plane printable characters
    high == 0 && (0x20..=0x7E).contains(&low) || low == b'\n' || low == b'\r' || low == b'\t'
}

Algorithm Pseudocode

for each even offset i in data:
    candidate_start = i
    char_count = 0
    printable_count = 0
    
    while i + 1 < data.len():
        low_byte = data[i]
        high_byte = data[i + 1]
        
        if is_printable_utf16le_char(low_byte, high_byte):
            char_count += 1
            printable_count += 1
            i += 2
        else if low_byte == 0 && high_byte == 0:
            # Null terminator found
            break
        else:
            # Invalid character, end of string
            break
    
    if char_count >= min_char_len:
        confidence = printable_count / char_count
        if confidence >= min_confidence:
            extract and add to results

Acceptance Criteria

✅ UTF-16LE strings are correctly extracted from Windows PE binaries
✅ Confidence scoring prevents false positives from random binary data
✅ Minimum length filtering is configurable and working
✅ Extracted strings include accurate offset, length, and encoding metadata
✅ Null termination is properly detected and handled
✅ Unit tests achieve >90% code coverage for the utf16.rs module
✅ Integration tests pass with real PE binary samples
✅ Documentation is updated in docs/src/string-extraction.md

Example Use Cases

VERSIONINFO Extraction from PE

$ stringy example.exe --encoding utf16le
[UTF-16LE] Microsoft Corporation
[UTF-16LE] Example Application
[UTF-16LE] Copyright © 2024

.NET String Literal Detection

$ stringy dotnet_app.dll --encoding utf16le --min-length 5
[UTF-16LE] Configuration string value
[UTF-16LE] Error message text

Dependencies

Blocked by: String Extraction Framework (basic extraction infrastructure must be in place)
Blocks: UTF-16BE String Extraction (can reuse confidence scoring logic)
Related: PE Resource Parsing (will benefit from UTF-16LE extraction)

References

Implementation algorithm: docs/src/string-extraction.md (UTF-16 Extraction section)
Type definitions: src/types.rs (Encoding::Utf16Le already defined)
Similar extraction: Study src/extraction/ascii.rs for pattern (when implemented)

Task Metadata

Component: stringy-analyzer
Module: extraction/utf16
Estimated Complexity: Medium
Estimated Time: 4-6 hours (implementation + tests)

Task-ID: stringy-analyzer/utf16le-string-extraction

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

✨ Issue enhanced with detailed technical context, implementation guidance, and acceptance criteria. Ready for implementation when the String Extraction Framework is complete.

@unclesp1d3r

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the current issue details
gh issue view 11 --json title,body,labels,assignees,author --repo EvilBit-Labs/StringyMcStringFace

Length of output: 1313

🏁 Script executed:

#!/bin/bash
# Get repository structure to understand the codebase
fd -t f -e rb -e yml -e yaml -e md | head -20

# Look for string handling or validation related code
fd -t f -e rb | xargs grep -l "String\|Validation\|validate" | head -10

Length of output: 404

🏁 Script executed:

#!/bin/bash
# Look at the Rust source structure
fd -t f -e rs | head -30

# Check if there's already an extraction module
fd -t d "extraction"

# Look at the string extraction documentation
cat docs/src/string-extraction.md

Length of output: 8592

🏁 Script executed:

#!/bin/bash
# Check README for project context
cat README.md | head -100

# Look at architecture documentation
cat docs/src/architecture.md | head -150

Length of output: 7019

🏁 Script executed:

#!/bin/bash
# Check the existing extraction module structure
cat src/extraction/mod.rs

# Look for any existing UTF-16 related code
rg -n "utf16|UTF16|Utf16" --type rust

# Check the types defined in the project
cat src/types.rs | head -200

Length of output: 5091

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced content
gh issue edit 11 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --title "Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering" \
  --body "## Summary

Implement robust UTF-16LE (Little Endian) string extraction capability to properly handle wide character strings commonly found in Windows PE binaries, .NET assemblies, and resource sections. This is a critical feature for analyzing Windows executables where UTF-16LE is the dominant string encoding.

## Context

UTF-16LE strings are ubiquitous in Windows binaries:
- **PE Resources**: VERSIONINFO, STRINGTABLE, RT_MANIFEST all use UTF-16LE
- **.NET Metadata**: String literals in managed assemblies
- **Windows API Strings**: Wide character (WCHAR) strings in native code
- **Registry Paths**: Many internal registry references

The standard \`strings\` command often mangles UTF-16 strings or extracts them as ASCII with null bytes interleaved, making them difficult to read and classify. Proper UTF-16LE extraction will significantly improve Stringy's effectiveness on Windows binaries.

## Technical Background

### UTF-16LE Detection Heuristics

UTF-16LE strings can be detected using the following heuristics:
1. **Even-length byte sequences** (UTF-16 uses 2 bytes per character)
2. **Low byte is printable ASCII** (0x20-0x7E)
3. **High byte is mostly zero** (for basic multilingual plane characters)
4. **Null termination pattern**: Two consecutive zero bytes (0x00 0x00)

### Confidence Scoring

To avoid false positives from coincidental byte patterns:
- **High confidence** (>90%): >90% printable characters with proper null termination
- **Medium confidence** (70-90%): >70% printable, reasonable length (>5 chars)
- **Low confidence** (<70%): >50% printable, may be coincidental data

## Requirements

### Core Functionality

- [ ] Create \`src/extraction/utf16.rs\` module with UTF-16LE extraction logic
- [ ] Implement \`extract_utf16le_strings(data: &[u8], config: &ExtractionConfig) -> Vec<RawString>\`
- [ ] Support configurable minimum character length (default: 3 characters = 6 bytes)
- [ ] Implement confidence scoring algorithm
- [ ] Handle proper null termination detection (0x00 0x00 sequence)
- [ ] Return strings with accurate offset, length, and encoding metadata

### Configuration

- [ ] Add \`min_utf16_len: usize\` to \`ExtractionConfig\` (default: 3)
- [ ] Add \`utf16_min_confidence: f32\` threshold (default: 0.7)
- [ ] Support section-aware minimum length overrides

### Integration

- [ ] Integrate UTF-16LE extraction into main extraction pipeline
- [ ] Wire up UTF-16LE extractor in \`src/extraction/mod.rs\`
- [ ] Ensure UTF-16LE strings are properly passed to classification module
- [ ] Add UTF-16LE support to deduplication logic

### Testing

- [ ] Unit tests for basic UTF-16LE string extraction
- [ ] Test cases for confidence scoring (high, medium, low)
- [ ] Test handling of null termination
- [ ] Test minimum length filtering
- [ ] Test false positive rejection (random binary data)
- [ ] Test with real-world Windows PE samples
- [ ] Edge cases: partial strings at section boundaries, interleaved data

## Proposed Implementation

### File Structure

\`\`\`rust
// src/extraction/utf16.rs

use crate::types::{Encoding, FoundString, StringSource};

/// Configuration for UTF-16LE extraction
pub struct Utf16Config {
    pub min_char_len: usize,
    pub min_confidence: f32,
}

/// Raw UTF-16LE string candidate with confidence score
struct Utf16Candidate {
    data: Vec<u8>,
    offset: usize,
    char_count: usize,
    confidence: f32,
}

/// Extract UTF-16LE strings from binary data
pub fn extract_utf16le_strings(
    data: &[u8],
    config: &Utf16Config,
    section_name: Option<String>,
) -> Vec<FoundString> {
    // Implementation based on algorithm in docs/src/string-extraction.md
    // 1. Scan for even-offset sequences with printable low bytes
    // 2. Calculate confidence scores
    // 3. Filter by minimum length and confidence
    // 4. Convert to FoundString with proper metadata
}

/// Calculate confidence score for a UTF-16LE candidate
fn calculate_confidence(data: &[u8]) -> f32 {
    // Percentage of valid printable UTF-16LE characters
    // Bonus for proper null termination
    // Penalty for excessive high bytes or invalid patterns
}

/// Check if a UTF-16LE character is printable
fn is_printable_utf16le_char(low: u8, high: u8) -> bool {
    // Basic multilingual plane printable characters
    high == 0 && (0x20..=0x7E).contains(&low) || low == b'\\n' || low == b'\\r' || low == b'\\t'
}
\`\`\`

### Algorithm Pseudocode

\`\`\`text
for each even offset i in data:
    candidate_start = i
    char_count = 0
    printable_count = 0
    
    while i + 1 < data.len():
        low_byte = data[i]
        high_byte = data[i + 1]
        
        if is_printable_utf16le_char(low_byte, high_byte):
            char_count += 1
            printable_count += 1
            i += 2
        else if low_byte == 0 && high_byte == 0:
            # Null terminator found
            break
        else:
            # Invalid character, end of string
            break
    
    if char_count >= min_char_len:
        confidence = printable_count / char_count
        if confidence >= min_confidence:
            extract and add to results
\`\`\`

## Acceptance Criteria

✅ UTF-16LE strings are correctly extracted from Windows PE binaries
✅ Confidence scoring prevents false positives from random binary data
✅ Minimum length filtering is configurable and working
✅ Extracted strings include accurate offset, length, and encoding metadata
✅ Null termination is properly detected and handled
✅ Unit tests achieve >90% code coverage for the utf16.rs module
✅ Integration tests pass with real PE binary samples
✅ Documentation is updated in \`docs/src/string-extraction.md\`

## Example Use Cases

### VERSIONINFO Extraction from PE
\`\`\`
$ stringy example.exe --encoding utf16le
[UTF-16LE] Microsoft Corporation
[UTF-16LE] Example Application
[UTF-16LE] Copyright © 2024
\`\`\`

### .NET String Literal Detection
\`\`\`
$ stringy dotnet_app.dll --encoding utf16le --min-length 5
[UTF-16LE] Configuration string value
[UTF-16LE] Error message text
\`\`\`

## Dependencies

- **Blocked by**: String Extraction Framework (basic extraction infrastructure must be in place)
- **Blocks**: UTF-16BE String Extraction (can reuse confidence scoring logic)
- **Related**: PE Resource Parsing (will benefit from UTF-16LE extraction)

## References

- Implementation algorithm: \`docs/src/string-extraction.md\` (UTF-16 Extraction section)
- Type definitions: \`src/types.rs\` (Encoding::Utf16Le already defined)
- Similar extraction: Study \`src/extraction/ascii.rs\` for pattern (when implemented)

## Task Metadata

- **Component**: stringy-analyzer
- **Module**: extraction/utf16
- **Estimated Complexity**: Medium
- **Estimated Time**: 4-6 hours (implementation + tests)

---
**Task-ID**: stringy-analyzer/utf16le-string-extraction"

# Also add a comment about the enhancement
gh issue comment 11 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --body "✨ Issue enhanced with detailed technical context, implementation guidance, and acceptance criteria. Ready for implementation when the String Extraction Framework is complete."

Length of output: 147

@unclesp1d3r I've successfully enhanced issue #11! Here's what was improved:

🎯 Title Enhancement

Before: "UTF-16LE String Extraction"
After: "Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering"

📝 Body Enhancements

Added Comprehensive Sections:

Context - Explains why UTF-16LE extraction is critical for Windows binaries
Technical Background - Detection heuristics and confidence scoring thresholds
Detailed Requirements - Broken down into Core Functionality, Configuration, Integration, and Testing
Proposed Implementation - Complete file structure with function signatures and algorithm pseudocode
Acceptance Criteria - Clear checkboxes for determining when the feature is complete
Example Use Cases - Real-world usage examples with expected output
Task Metadata - Complexity and time estimates

Key Technical Details Added:

UTF-16LE detection algorithm based on the project's documentation
Confidence scoring thresholds (high >90%, medium 70-90%, low <70%)
Default configuration values (min 3 characters, 0.7 confidence threshold)
Complete Rust code structure template for src/extraction/utf16.rs
Comprehensive testing requirements including edge cases

🏷️ Metadata Status

The existing labels are already appropriate and remain unchanged:

✅ area:analyzer - Correct component
✅ type:enhancement - Appropriate type
✅ status:backlog - Current status
✅ lang:rust - Implementation language
✅ needs:tests - Testing requirement flagged

Assignment to you (@unclesp1d3r) is maintained.

The issue is now ready for implementation with clear guidance, acceptance criteria, and technical specifications! 🚀

@traycerai generate

@unclesp1d3r

Hi @unclesp1d3r, I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The codebase has a well-structured extraction framework with:

Existing Infrastructure: ASCII extractor (src/extraction/ascii.rs) serves as the reference implementation with comprehensive testing and noise filtering integration.

Configuration Ready: ExtractionConfig already has min_wide_length field (default: 3) for UTF-16 strings, and the Encoding::Utf16Le enum variant exists in types.rs.

Noise Filtering: The CompositeNoiseFilter in src/extraction/filters.rs operates on decoded UTF-8 text (the text field of FoundString), so it will work seamlessly with UTF-16LE strings after decoding.

UTF-16LE Decoding: The decode_utf16le function already exists in src/extraction/pe_resources.rs and can be reused or adapted for the new UTF-16LE extractor.

Integration Pattern: The BasicExtractor::extract_from_section method in src/extraction/mod.rs orchestrates extraction by calling specialized extractors (ASCII, UTF-8) and applying noise filtering with confidence thresholds.

Testing Pattern: Comprehensive unit tests in tests/test_ascii_extraction.rs provide a clear template for UTF-16LE tests.

Approach

Create a new src/extraction/utf16.rs module implementing UTF-16LE string extraction with confidence scoring, following the established ASCII extractor pattern. The module will provide extract_utf16le_strings() for raw extraction and extract_from_section() for section-aware extraction with metadata population. Integration into BasicExtractor::extract_from_section() will check for Encoding::Utf16Le in enabled_encodings and apply the same noise filtering pipeline used for ASCII/UTF-8. Configuration will use the existing min_wide_length field and add utf16_min_confidence to ExtractionConfig. Comprehensive unit tests will cover basic extraction, confidence scoring, null termination, edge cases, and integration with section metadata.

Reasoning

I explored the codebase structure by listing directories, reading key files including type definitions, extraction modules (ASCII, filters, PE resources), configuration structures, and test files. I examined the existing ASCII extractor pattern, noise filtering infrastructure, UTF-16LE decoding in PE resources, and the BasicExtractor integration point. I searched for UTF-16 references to understand current usage and verified the configuration fields and dependencies in Cargo.toml.

Proposed File Changes

📄 src/extraction/utf16.rs ^(NEW) 🔗

References

📄 src/extraction/ascii.rs

📄 src/extraction/pe_resources.rs

📄 src/types.rs

📄 src/extraction/config.rs

📄 src/extraction/filters.rs

Create a new UTF-16LE string extraction module following the pattern established in src/extraction/ascii.rs.

Module Structure:

Configuration Struct: Define Utf16ExtractionConfig with fields:

min_char_len: usize (minimum character count, default: 3)

max_char_len: Option<usize> (maximum character count, default: None)

min_confidence: f32 (minimum confidence threshold, default: 0.7)

Helper Functions:

is_printable_utf16le_char(low: u8, high: u8) -> bool: Check if a UTF-16LE character is printable (high byte is 0 and low byte is in 0x20-0x7E range, or common whitespace like tab/newline/carriage return)

decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()>: Convert UTF-16LE byte sequence to UTF-8 String using u16::from_le_bytes and String::from_utf16, similar to the implementation in src/extraction/pe_resources.rs but adapted for extraction use case

calculate_confidence(data: &[u8], char_count: usize, has_null_terminator: bool) -> f32: Calculate confidence score based on percentage of printable characters, proper null termination (bonus), and character count

Core Extraction Function: extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>:

Scan through data at even offsets (UTF-16LE requires 2-byte alignment)

For each position, accumulate printable UTF-16LE characters

Detect null termination (0x00 0x00 sequence)

Calculate confidence score for each candidate

Filter by minimum character length and confidence threshold

Convert to FoundString with Encoding::Utf16Le, proper offset, length (in bytes), and confidence score

Handle edge cases: partial strings at buffer boundaries, strings at start/end of data

Section-Aware Extraction: extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>:

Extract section data slice with bounds checking (similar to src/extraction/ascii.rs)

Call extract_utf16le_strings() on section data

Build FilterContext from section metadata

Apply noise filtering if enabled using CompositeNoiseFilter

Adjust offsets to absolute positions (section.offset + relative_offset)

Populate section name and calculate RVA if available

Filter by confidence threshold

Implementation Details:

Follow the algorithm pseudocode from the GitHub issue and docs/src/string-extraction.md (lines 238-274)

Use the same error handling and bounds checking patterns as src/extraction/ascii.rs

Ensure confidence scoring considers: printable character percentage, null termination presence, and string length

High confidence (>0.9): >90% printable with proper null termination

Medium confidence (0.7-0.9): >70% printable, reasonable length

Low confidence (<0.7): >50% printable, may be coincidental

Documentation:

Add comprehensive module-level documentation explaining UTF-16LE extraction, confidence scoring, and usage examples

Document all public functions with examples

Include notes on Windows PE binary string extraction use cases

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

📄 src/types.rs

Integrate UTF-16LE extraction into the main extraction pipeline.

Changes Required:

Module Declaration (after line 110):

Add pub mod utf16; to declare the new UTF-16LE extraction module

Add public re-export: pub use utf16::{extract_utf16le_strings, Utf16ExtractionConfig}; in the re-exports section (around line 110)

ExtractionConfig Enhancement (around lines 133-183):

Add new field utf16_min_confidence: f32 with default value 0.7 in the ExtractionConfig struct (around line 159)

Update the Default implementation to include utf16_min_confidence: 0.7 (around line 180)

Add validation in ExtractionConfig::validate() method to ensure utf16_min_confidence is between 0.0 and 1.0 (around line 205)

BasicExtractor Integration (in extract_from_section method, around lines 416-537):

After UTF-8 extraction block (around line 534), add UTF-16LE extraction logic:

Check if Encoding::Utf16Le is in config.encodings or config.enabled_encodings

Create Utf16ExtractionConfig using config.min_wide_length and config.utf16_min_confidence

Call utf16::extract_from_section() with section, data, config, noise filter config, and thresholds

Extend found_strings with the UTF-16LE results

Follow the same pattern as UTF-8 extraction: build filter context, apply noise filtering if enabled, filter by confidence threshold

Documentation Updates:

Update module-level documentation (lines 1-94) to mention UTF-16LE extraction support

Add UTF-16LE to the list of supported encodings in the module documentation

Integration Pattern:
The UTF-16LE extraction should be added after the UTF-8 extraction block and before the Ok(found_strings) return statement. It should reuse the same noise_filter_config and filter instances created for ASCII/UTF-8 extraction to maintain consistency and avoid redundant filter creation.

📄 src/lib.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update library-level documentation and re-exports to include UTF-16LE extraction support.

Changes Required:

Documentation Update (around lines 52-60):

Update the extraction module description to mention UTF-16LE support alongside ASCII and PE resources

Change the status indicator from "ASCII extraction and PE resources complete" to "ASCII, UTF-16LE extraction and PE resources complete"

Re-exports Update (around line 76):

Add Utf16ExtractionConfig to the re-exports from the extraction module

Update the re-export line to: pub use extraction::{AsciiExtractionConfig, Utf16ExtractionConfig, BasicExtractor, ExtractionConfig, StringExtractor};

Purpose:
These changes ensure that users of the library can easily access the new UTF-16LE extraction functionality through the top-level API without needing to navigate into the extraction submodule.

📄 tests/test_utf16_extraction.rs ^(NEW) 🔗

References

📄 tests/test_ascii_extraction.rs

📄 src/extraction/utf16.rs ^(NEW)

📄 src/types.rs

Create comprehensive unit tests for UTF-16LE string extraction, following the pattern established in tests/test_ascii_extraction.rs.

Test Coverage:

Basic Extraction Tests:

test_basic_utf16le_extraction: Extract simple UTF-16LE strings with null terminators

test_minimum_character_length: Verify minimum character length filtering (default 3 characters = 6 bytes)

test_custom_minimum_length: Test custom minimum length configuration

test_null_terminated_strings: Verify proper null termination detection (0x00 0x00)

Confidence Scoring Tests:

test_high_confidence_strings: Strings with >90% printable characters and null termination

test_medium_confidence_strings: Strings with 70-90% printable characters

test_low_confidence_strings: Strings with 50-70% printable characters

test_confidence_threshold_filtering: Verify strings below threshold are filtered out

Edge Cases:

test_empty_input: Empty data returns empty vector

test_no_valid_strings: Binary data with no valid UTF-16LE sequences

test_string_at_start: String at buffer start (offset 0)

test_string_at_end: String at buffer end without null terminator

test_odd_length_data: Handle odd-length data gracefully

test_partial_string_at_boundary: Partial UTF-16LE character at section boundary

test_interleaved_binary_data: UTF-16LE strings mixed with binary data

Section Integration Tests:

test_section_metadata_attachment: Verify section name, offset, and RVA are correctly populated

test_section_boundary_handling: Ensure extraction respects section boundaries

test_section_out_of_bounds: Handle sections beyond data bounds

test_different_section_types: Test extraction from various section types (.rdata, .data, .rsrc)

Real-World Scenarios:

test_windows_pe_versioninfo: Simulate VERSIONINFO resource strings (e.g., "Microsoft Corporation", "Copyright © 2024")

test_dotnet_string_literals: Simulate .NET assembly string literals

test_registry_paths: Test Windows registry path extraction

Noise Filtering Integration:

test_noise_filtering_enabled: Verify noise filtering reduces confidence for noisy strings

test_noise_filtering_disabled: Verify all strings extracted when filtering disabled

test_confidence_threshold_application: Test various confidence thresholds

Configuration Tests:

test_config_defaults: Verify default configuration values

test_config_customization: Test custom min_char_len, max_char_len, and min_confidence

test_max_length_filtering: Verify maximum length filtering works correctly

Test Data Helpers:

Create helper functions to generate UTF-16LE test data:

create_utf16le_string(text: &str) -> Vec<u8>: Convert ASCII string to UTF-16LE bytes

create_test_section(name: &str, offset: u64, size: u64, rva: Option<u64>) -> SectionInfo: Create test section metadata

create_utf16le_with_confidence(text: &str, noise_ratio: f32) -> Vec<u8>: Create UTF-16LE data with controlled noise for confidence testing

Assertions:

Verify encoding field is Encoding::Utf16Le

Verify source field is StringSource::SectionData

Verify confidence scores match expected ranges

Verify offset, length, section, and rva fields are correctly populated

Verify string text is correctly decoded to UTF-8

📄 tests/integration_extraction.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

📄 tests/test_ascii_extraction.rs

Add integration tests for UTF-16LE extraction to verify end-to-end functionality with the BasicExtractor.

New Test Cases:

test_utf16le_extraction_with_basic_extractor:

Create a mock binary with UTF-16LE strings in a .rdata section

Use BasicExtractor with ExtractionConfig that includes Encoding::Utf16Le in enabled_encodings

Verify UTF-16LE strings are extracted with correct metadata

Verify confidence scores are applied

test_utf16le_encoding_filtering:

Test that UTF-16LE strings are only extracted when Encoding::Utf16Le is in enabled_encodings

Verify that disabling UTF-16LE encoding prevents extraction

test_utf16le_with_noise_filtering:

Create test data with both high-quality and noisy UTF-16LE strings

Enable noise filtering and verify low-confidence strings are filtered out

Disable noise filtering and verify all strings are extracted

test_utf16le_min_wide_length_config:

Test that min_wide_length configuration is respected

Verify strings shorter than min_wide_length are filtered out

test_utf16le_confidence_threshold:

Test various utf16_min_confidence threshold values

Verify strings below threshold are filtered out

Integration with Existing Tests:

Add UTF-16LE test cases to existing integration test functions if applicable

Ensure UTF-16LE extraction works alongside ASCII and UTF-8 extraction

Verify no conflicts or interference between different encoding extractors

Test Data:

Use realistic UTF-16LE strings from Windows PE binaries (version info, registry paths, error messages)

Include edge cases like very short strings, very long strings, and strings with special characters
📄 docs/src/string-extraction.md ^(MODIFY) 🔗
References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update the string extraction documentation to reflect the completed UTF-16LE implementation.

Changes Required:
UTF-16 Extraction Section (lines 212-283):

Update the section header to indicate UTF-16LE is now implemented (change from future tense to present tense)

Add a note that UTF-16LE extraction is complete and available

Update the implementation strategy section to reference the actual implementation in src/extraction/utf16.rs

Add usage examples showing how to use extract_utf16le_strings() and Utf16ExtractionConfig
Configuration Section (lines 404-418):
Update the ExtractionConfig documentation to include the new utf16_min_confidence field
Add example showing how to configure UTF-16LE extraction:
let mut config = ExtractionConfig::default();
config.min_wide_length = 3;
config.utf16_min_confidence = 0.7;
config.enabled_encodings.push(Encoding::Utf16Le);
Encoding Support Section (lines 11-87):

Update the encoding support list to show UTF-16LE as implemented (✅)

Add a brief description of UTF-16LE extraction capabilities

Examples Section:

Add practical examples of UTF-16LE extraction from Windows PE binaries:

Extracting VERSIONINFO strings

Extracting .NET string literals

Extracting registry paths

Include code snippets showing configuration and usage

Confidence Scoring Section (lines 276-282):

Expand the confidence scoring explanation for UTF-16LE:

Explain how printable character percentage is calculated

Describe the null termination bonus

Provide examples of high, medium, and low confidence strings
Documentation Style:

Maintain consistency with existing documentation style

Use clear, concise language

Include practical examples and use cases

Reference the actual implementation files for developers who want to dive deeper
Diagram
sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module as utf16.rs
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if Utf16Le in enabled_encodings
    
    alt UTF-16LE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, config)
        Utf16Module->>Utf16Module: Extract section data slice
        Utf16Module->>Utf16Module: extract_utf16le_strings(data, config)
        
        loop For each even offset
            Utf16Module->>Utf16Module: Scan for printable UTF-16LE chars
            Utf16Module->>Utf16Module: Detect null termination (0x00 0x00)
            Utf16Module->>Utf16Module: Calculate confidence score
            
            alt Meets min_char_len & min_confidence
                Utf16Module->>Utf16Module: Decode to UTF-8 String
                Utf16Module->>FoundString: Create with Encoding::Utf16Le
            end
        end
        
        Utf16Module->>NoiseFilter: calculate_confidence(text, context)
        NoiseFilter-->>Utf16Module: confidence score
        
        alt confidence >= threshold
            Utf16Module->>Utf16Module: Adjust offsets, populate metadata
            Utf16Module-->>BasicExtractor: Vec<FoundString>
        end
        
        BasicExtractor->>BasicExtractor: Extend found_strings
    end
    
    BasicExtractor-->>User: All extracted strings
Loading
Import In IDE
🤖 Prompt for AI Agents
I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-structured extraction framework with:

1. **Existing Infrastructure**: ASCII extractor (`src/extraction/ascii.rs`) serves as the reference implementation with comprehensive testing and noise filtering integration.

2. **Configuration Ready**: `ExtractionConfig` already has `min_wide_length` field (default: 3) for UTF-16 strings, and the `Encoding::Utf16Le` enum variant exists in `types.rs`.

3. **Noise Filtering**: The `CompositeNoiseFilter` in `src/extraction/filters.rs` operates on decoded UTF-8 text (the `text` field of `FoundString`), so it will work seamlessly with UTF-16LE strings after decoding.

4. **UTF-16LE Decoding**: The `decode_utf16le` function already exists in `src/extraction/pe_resources.rs` and can be reused or adapted for the new UTF-16LE extractor.

5. **Integration Pattern**: The `BasicExtractor::extract_from_section` method in `src/extraction/mod.rs` orchestrates extraction by calling specialized extractors (ASCII, UTF-8) and applying noise filtering with confidence thresholds.

6. **Testing Pattern**: Comprehensive unit tests in `tests/test_ascii_extraction.rs` provide a clear template for UTF-16LE tests.

### Approach

Create a new `src/extraction/utf16.rs` module implementing UTF-16LE string extraction with confidence scoring, following the established ASCII extractor pattern. The module will provide `extract_utf16le_strings()` for raw extraction and `extract_from_section()` for section-aware extraction with metadata population. Integration into `BasicExtractor::extract_from_section()` will check for `Encoding::Utf16Le` in `enabled_encodings` and apply the same noise filtering pipeline used for ASCII/UTF-8. Configuration will use the existing `min_wide_length` field and add `utf16_min_confidence` to `ExtractionConfig`. Comprehensive unit tests will cover basic extraction, confidence scoring, null termination, edge cases, and integration with section metadata.

### Reasoning

I explored the codebase structure by listing directories, reading key files including type definitions, extraction modules (ASCII, filters, PE resources), configuration structures, and test files. I examined the existing ASCII extractor pattern, noise filtering infrastructure, UTF-16LE decoding in PE resources, and the BasicExtractor integration point. I searched for UTF-16 references to understand current usage and verified the configuration fields and dependencies in Cargo.toml.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module as utf16.rs
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if Utf16Le in enabled_encodings
    
    alt UTF-16LE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, config)
        Utf16Module->>Utf16Module: Extract section data slice
        Utf16Module->>Utf16Module: extract_utf16le_strings(data, config)
        
        loop For each even offset
            Utf16Module->>Utf16Module: Scan for printable UTF-16LE chars
            Utf16Module->>Utf16Module: Detect null termination (0x00 0x00)
            Utf16Module->>Utf16Module: Calculate confidence score
            
            alt Meets min_char_len & min_confidence
                Utf16Module->>Utf16Module: Decode to UTF-8 String
                Utf16Module->>FoundString: Create with Encoding::Utf16Le
            end
        end
        
        Utf16Module->>NoiseFilter: calculate_confidence(text, context)
        NoiseFilter-->>Utf16Module: confidence score
        
        alt confidence >= threshold
            Utf16Module->>Utf16Module: Adjust offsets, populate metadata
            Utf16Module-->>BasicExtractor: Vec<FoundString>
        end
        
        BasicExtractor->>BasicExtractor: Extend found_strings
    end
    
    BasicExtractor-->>User: All extracted strings

## Proposed File Changes

### src/extraction/utf16.rs(NEW)

References: 

- src/extraction/ascii.rs
- src/extraction/pe_resources.rs
- src/types.rs
- src/extraction/config.rs
- src/extraction/filters.rs

Create a new UTF-16LE string extraction module following the pattern established in `src/extraction/ascii.rs`.

**Module Structure:**

1. **Configuration Struct**: Define `Utf16ExtractionConfig` with fields:
   - `min_char_len: usize` (minimum character count, default: 3)
   - `max_char_len: Option<usize>` (maximum character count, default: None)
   - `min_confidence: f32` (minimum confidence threshold, default: 0.7)

2. **Helper Functions**:
   - `is_printable_utf16le_char(low: u8, high: u8) -> bool`: Check if a UTF-16LE character is printable (high byte is 0 and low byte is in 0x20-0x7E range, or common whitespace like tab/newline/carriage return)
   - `decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()>`: Convert UTF-16LE byte sequence to UTF-8 String using `u16::from_le_bytes` and `String::from_utf16`, similar to the implementation in `src/extraction/pe_resources.rs` but adapted for extraction use case
   - `calculate_confidence(data: &[u8], char_count: usize, has_null_terminator: bool) -> f32`: Calculate confidence score based on percentage of printable characters, proper null termination (bonus), and character count

3. **Core Extraction Function**: `extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>`:
   - Scan through data at even offsets (UTF-16LE requires 2-byte alignment)
   - For each position, accumulate printable UTF-16LE characters
   - Detect null termination (0x00 0x00 sequence)
   - Calculate confidence score for each candidate
   - Filter by minimum character length and confidence threshold
   - Convert to `FoundString` with `Encoding::Utf16Le`, proper offset, length (in bytes), and confidence score
   - Handle edge cases: partial strings at buffer boundaries, strings at start/end of data

4. **Section-Aware Extraction**: `extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>`:
   - Extract section data slice with bounds checking (similar to `src/extraction/ascii.rs`)
   - Call `extract_utf16le_strings()` on section data
   - Build `FilterContext` from section metadata
   - Apply noise filtering if enabled using `CompositeNoiseFilter`
   - Adjust offsets to absolute positions (section.offset + relative_offset)
   - Populate section name and calculate RVA if available
   - Filter by confidence threshold

**Implementation Details:**
- Follow the algorithm pseudocode from the GitHub issue and `docs/src/string-extraction.md` (lines 238-274)
- Use the same error handling and bounds checking patterns as `src/extraction/ascii.rs`
- Ensure confidence scoring considers: printable character percentage, null termination presence, and string length
- High confidence (>0.9): >90% printable with proper null termination
- Medium confidence (0.7-0.9): >70% printable, reasonable length
- Low confidence (<0.7): >50% printable, may be coincidental

**Documentation:**
- Add comprehensive module-level documentation explaining UTF-16LE extraction, confidence scoring, and usage examples
- Document all public functions with examples
- Include notes on Windows PE binary string extraction use cases

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs
- src/types.rs

Integrate UTF-16LE extraction into the main extraction pipeline.

**Changes Required:**

1. **Module Declaration** (after line 110):
   - Add `pub mod utf16;` to declare the new UTF-16LE extraction module
   - Add public re-export: `pub use utf16::{extract_utf16le_strings, Utf16ExtractionConfig};` in the re-exports section (around line 110)

2. **ExtractionConfig Enhancement** (around lines 133-183):
   - Add new field `utf16_min_confidence: f32` with default value 0.7 in the `ExtractionConfig` struct (around line 159)
   - Update the `Default` implementation to include `utf16_min_confidence: 0.7` (around line 180)
   - Add validation in `ExtractionConfig::validate()` method to ensure `utf16_min_confidence` is between 0.0 and 1.0 (around line 205)

3. **BasicExtractor Integration** (in `extract_from_section` method, around lines 416-537):
   - After UTF-8 extraction block (around line 534), add UTF-16LE extraction logic:
     - Check if `Encoding::Utf16Le` is in `config.encodings` or `config.enabled_encodings`
     - Create `Utf16ExtractionConfig` using `config.min_wide_length` and `config.utf16_min_confidence`
     - Call `utf16::extract_from_section()` with section, data, config, noise filter config, and thresholds
     - Extend `found_strings` with the UTF-16LE results
   - Follow the same pattern as UTF-8 extraction: build filter context, apply noise filtering if enabled, filter by confidence threshold

4. **Documentation Updates**:
   - Update module-level documentation (lines 1-94) to mention UTF-16LE extraction support
   - Add UTF-16LE to the list of supported encodings in the module documentation

**Integration Pattern:**
The UTF-16LE extraction should be added after the UTF-8 extraction block and before the `Ok(found_strings)` return statement. It should reuse the same `noise_filter_config` and `filter` instances created for ASCII/UTF-8 extraction to maintain consistency and avoid redundant filter creation.

### src/lib.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update library-level documentation and re-exports to include UTF-16LE extraction support.

**Changes Required:**

1. **Documentation Update** (around lines 52-60):
   - Update the extraction module description to mention UTF-16LE support alongside ASCII and PE resources
   - Change the status indicator from "ASCII extraction and PE resources complete" to "ASCII, UTF-16LE extraction and PE resources complete"

2. **Re-exports Update** (around line 76):
   - Add `Utf16ExtractionConfig` to the re-exports from the extraction module
   - Update the re-export line to: `pub use extraction::{AsciiExtractionConfig, Utf16ExtractionConfig, BasicExtractor, ExtractionConfig, StringExtractor};`

**Purpose:**
These changes ensure that users of the library can easily access the new UTF-16LE extraction functionality through the top-level API without needing to navigate into the extraction submodule.

### tests/test_utf16_extraction.rs(NEW)

References: 

- tests/test_ascii_extraction.rs
- src/extraction/utf16.rs(NEW)
- src/types.rs

Create comprehensive unit tests for UTF-16LE string extraction, following the pattern established in `tests/test_ascii_extraction.rs`.

**Test Coverage:**

1. **Basic Extraction Tests**:
   - `test_basic_utf16le_extraction`: Extract simple UTF-16LE strings with null terminators
   - `test_minimum_character_length`: Verify minimum character length filtering (default 3 characters = 6 bytes)
   - `test_custom_minimum_length`: Test custom minimum length configuration
   - `test_null_terminated_strings`: Verify proper null termination detection (0x00 0x00)

2. **Confidence Scoring Tests**:
   - `test_high_confidence_strings`: Strings with >90% printable characters and null termination
   - `test_medium_confidence_strings`: Strings with 70-90% printable characters
   - `test_low_confidence_strings`: Strings with 50-70% printable characters
   - `test_confidence_threshold_filtering`: Verify strings below threshold are filtered out

3. **Edge Cases**:
   - `test_empty_input`: Empty data returns empty vector
   - `test_no_valid_strings`: Binary data with no valid UTF-16LE sequences
   - `test_string_at_start`: String at buffer start (offset 0)
   - `test_string_at_end`: String at buffer end without null terminator
   - `test_odd_length_data`: Handle odd-length data gracefully
   - `test_partial_string_at_boundary`: Partial UTF-16LE character at section boundary
   - `test_interleaved_binary_data`: UTF-16LE strings mixed with binary data

4. **Section Integration Tests**:
   - `test_section_metadata_attachment`: Verify section name, offset, and RVA are correctly populated
   - `test_section_boundary_handling`: Ensure extraction respects section boundaries
   - `test_section_out_of_bounds`: Handle sections beyond data bounds
   - `test_different_section_types`: Test extraction from various section types (.rdata, .data, .rsrc)

5. **Real-World Scenarios**:
   - `test_windows_pe_versioninfo`: Simulate VERSIONINFO resource strings (e.g., "Microsoft Corporation", "Copyright © 2024")
   - `test_dotnet_string_literals`: Simulate .NET assembly string literals
   - `test_registry_paths`: Test Windows registry path extraction

6. **Noise Filtering Integration**:
   - `test_noise_filtering_enabled`: Verify noise filtering reduces confidence for noisy strings
   - `test_noise_filtering_disabled`: Verify all strings extracted when filtering disabled
   - `test_confidence_threshold_application`: Test various confidence thresholds

7. **Configuration Tests**:
   - `test_config_defaults`: Verify default configuration values
   - `test_config_customization`: Test custom min_char_len, max_char_len, and min_confidence
   - `test_max_length_filtering`: Verify maximum length filtering works correctly

**Test Data Helpers**:
- Create helper functions to generate UTF-16LE test data:
  - `create_utf16le_string(text: &str) -> Vec<u8>`: Convert ASCII string to UTF-16LE bytes
  - `create_test_section(name: &str, offset: u64, size: u64, rva: Option<u64>) -> SectionInfo`: Create test section metadata
  - `create_utf16le_with_confidence(text: &str, noise_ratio: f32) -> Vec<u8>`: Create UTF-16LE data with controlled noise for confidence testing

**Assertions:**
- Verify `encoding` field is `Encoding::Utf16Le`
- Verify `source` field is `StringSource::SectionData`
- Verify `confidence` scores match expected ranges
- Verify `offset`, `length`, `section`, and `rva` fields are correctly populated
- Verify string text is correctly decoded to UTF-8

### tests/integration_extraction.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)
- tests/test_ascii_extraction.rs

Add integration tests for UTF-16LE extraction to verify end-to-end functionality with the `BasicExtractor`.

**New Test Cases:**

1. **test_utf16le_extraction_with_basic_extractor**:
   - Create a mock binary with UTF-16LE strings in a .rdata section
   - Use `BasicExtractor` with `ExtractionConfig` that includes `Encoding::Utf16Le` in `enabled_encodings`
   - Verify UTF-16LE strings are extracted with correct metadata
   - Verify confidence scores are applied

2. **test_utf16le_encoding_filtering**:
   - Test that UTF-16LE strings are only extracted when `Encoding::Utf16Le` is in `enabled_encodings`
   - Verify that disabling UTF-16LE encoding prevents extraction

3. **test_utf16le_with_noise_filtering**:
   - Create test data with both high-quality and noisy UTF-16LE strings
   - Enable noise filtering and verify low-confidence strings are filtered out
   - Disable noise filtering and verify all strings are extracted

4. **test_utf16le_min_wide_length_config**:
   - Test that `min_wide_length` configuration is respected
   - Verify strings shorter than `min_wide_length` are filtered out

5. **test_utf16le_confidence_threshold**:
   - Test various `utf16_min_confidence` threshold values
   - Verify strings below threshold are filtered out

**Integration with Existing Tests:**
- Add UTF-16LE test cases to existing integration test functions if applicable
- Ensure UTF-16LE extraction works alongside ASCII and UTF-8 extraction
- Verify no conflicts or interference between different encoding extractors

**Test Data:**
- Use realistic UTF-16LE strings from Windows PE binaries (version info, registry paths, error messages)
- Include edge cases like very short strings, very long strings, and strings with special characters

### docs/src/string-extraction.md(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update the string extraction documentation to reflect the completed UTF-16LE implementation.

**Changes Required:**

1. **UTF-16 Extraction Section** (lines 212-283):
   - Update the section header to indicate UTF-16LE is now implemented (change from future tense to present tense)
   - Add a note that UTF-16LE extraction is complete and available
   - Update the implementation strategy section to reference the actual implementation in `src/extraction/utf16.rs`
   - Add usage examples showing how to use `extract_utf16le_strings()` and `Utf16ExtractionConfig`

2. **Configuration Section** (lines 404-418):
   - Update the `ExtractionConfig` documentation to include the new `utf16_min_confidence` field
   - Add example showing how to configure UTF-16LE extraction:
     ```rust
     let mut config = ExtractionConfig::default();
     config.min_wide_length = 3;
     config.utf16_min_confidence = 0.7;
     config.enabled_encodings.push(Encoding::Utf16Le);
     ```

3. **Encoding Support Section** (lines 11-87):
   - Update the encoding support list to show UTF-16LE as implemented (✅)
   - Add a brief description of UTF-16LE extraction capabilities

4. **Examples Section**:
   - Add practical examples of UTF-16LE extraction from Windows PE binaries:
     - Extracting VERSIONINFO strings
     - Extracting .NET string literals
     - Extracting registry paths
   - Include code snippets showing configuration and usage

5. **Confidence Scoring Section** (lines 276-282):
   - Expand the confidence scoring explanation for UTF-16LE:
     - Explain how printable character percentage is calculated
     - Describe the null termination bonus
     - Provide examples of high, medium, and low confidence strings

**Documentation Style:**
- Maintain consistency with existing documentation style
- Use clear, concise language
- Include practical examples and use cases
- Reference the actual implementation files for developers who want to dive deeper
Developer Humor

🎯 UTF-16LE strings, once hidden in plain sight,\n> Now extracted with confidence, shining bright!\n> From Windows PEs to .NET's embrace,\n> Wide characters find their rightful place.\n> 🔍 Two bytes per char, but worth the space! 💾

Execution Information

Branch: main
Commit: c50e9b6

Plan

Observations

The codebase has a well-structured extraction framework with:

Existing Infrastructure: ASCII extractor (src/extraction/ascii.rs) serves as the reference implementation with comprehensive testing and noise filtering integration.
Configuration Ready: ExtractionConfig already has min_wide_length field (default: 3) for UTF-16 strings, and the Encoding::Utf16Le enum variant exists in types.rs.
Noise Filtering: The CompositeNoiseFilter in src/extraction/filters.rs operates on decoded UTF-8 text (the text field of FoundString), so it will work seamlessly with UTF-16LE strings after decoding.
UTF-16LE Decoding: The decode_utf16le function already exists in src/extraction/pe_resources.rs and can be reused or adapted for the new UTF-16LE extractor.
Integration Pattern: The BasicExtractor::extract_from_section method in src/extraction/mod.rs orchestrates extraction by calling specialized extractors (ASCII, UTF-8) and applying noise filtering with confidence thresholds.
Testing Pattern: Comprehensive unit tests in tests/test_ascii_extraction.rs provide a clear template for UTF-16LE tests.

Approach

Create a new src/extraction/utf16.rs module implementing UTF-16LE string extraction with confidence scoring, following the established ASCII extractor pattern. The module will provide extract_utf16le_strings() for raw extraction and extract_from_section() for section-aware extraction with metadata population. Integration into BasicExtractor::extract_from_section() will check for Encoding::Utf16Le in enabled_encodings and apply the same noise filtering pipeline used for ASCII/UTF-8. Configuration will use the existing min_wide_length field and add utf16_min_confidence to ExtractionConfig. Comprehensive unit tests will cover basic extraction, confidence scoring, null termination, edge cases, and integration with section metadata.

Reasoning

I explored the codebase structure by listing directories, reading key files including type definitions, extraction modules (ASCII, filters, PE resources), configuration structures, and test files. I examined the existing ASCII extractor pattern, noise filtering infrastructure, UTF-16LE decoding in PE resources, and the BasicExtractor integration point. I searched for UTF-16 references to understand current usage and verified the configuration fields and dependencies in Cargo.toml.

Proposed File Changes

📄 src/extraction/utf16.rs ^(NEW) 🔗

References

📄 src/extraction/ascii.rs

📄 src/extraction/pe_resources.rs

📄 src/types.rs

📄 src/extraction/config.rs

📄 src/extraction/filters.rs

Create a new UTF-16LE string extraction module following the pattern established in src/extraction/ascii.rs.

Module Structure:

Configuration Struct: Define Utf16ExtractionConfig with fields:

min_char_len: usize (minimum character count, default: 3)

max_char_len: Option<usize> (maximum character count, default: None)

min_confidence: f32 (minimum confidence threshold, default: 0.7)

Helper Functions:

is_printable_utf16le_char(low: u8, high: u8) -> bool: Check if a UTF-16LE character is printable (high byte is 0 and low byte is in 0x20-0x7E range, or common whitespace like tab/newline/carriage return)

decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()>: Convert UTF-16LE byte sequence to UTF-8 String using u16::from_le_bytes and String::from_utf16, similar to the implementation in src/extraction/pe_resources.rs but adapted for extraction use case

calculate_confidence(data: &[u8], char_count: usize, has_null_terminator: bool) -> f32: Calculate confidence score based on percentage of printable characters, proper null termination (bonus), and character count

Core Extraction Function: extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>:

Scan through data at even offsets (UTF-16LE requires 2-byte alignment)

For each position, accumulate printable UTF-16LE characters

Detect null termination (0x00 0x00 sequence)

Calculate confidence score for each candidate

Filter by minimum character length and confidence threshold

Convert to FoundString with Encoding::Utf16Le, proper offset, length (in bytes), and confidence score

Handle edge cases: partial strings at buffer boundaries, strings at start/end of data

Section-Aware Extraction: extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>:

Extract section data slice with bounds checking (similar to src/extraction/ascii.rs)

Call extract_utf16le_strings() on section data

Build FilterContext from section metadata

Apply noise filtering if enabled using CompositeNoiseFilter

Adjust offsets to absolute positions (section.offset + relative_offset)

Populate section name and calculate RVA if available

Filter by confidence threshold

Implementation Details:

Follow the algorithm pseudocode from the GitHub issue and docs/src/string-extraction.md (lines 238-274)

Use the same error handling and bounds checking patterns as src/extraction/ascii.rs

Ensure confidence scoring considers: printable character percentage, null termination presence, and string length

High confidence (>0.9): >90% printable with proper null termination

Medium confidence (0.7-0.9): >70% printable, reasonable length

Low confidence (<0.7): >50% printable, may be coincidental

Documentation:

Add comprehensive module-level documentation explaining UTF-16LE extraction, confidence scoring, and usage examples

Document all public functions with examples

Include notes on Windows PE binary string extraction use cases

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/ascii.rs

📄 src/types.rs

Integrate UTF-16LE extraction into the main extraction pipeline.

Changes Required:

Module Declaration (after line 110):

Add pub mod utf16; to declare the new UTF-16LE extraction module

Add public re-export: pub use utf16::{extract_utf16le_strings, Utf16ExtractionConfig}; in the re-exports section (around line 110)

ExtractionConfig Enhancement (around lines 133-183):

Add new field utf16_min_confidence: f32 with default value 0.7 in the ExtractionConfig struct (around line 159)

Update the Default implementation to include utf16_min_confidence: 0.7 (around line 180)

Add validation in ExtractionConfig::validate() method to ensure utf16_min_confidence is between 0.0 and 1.0 (around line 205)

BasicExtractor Integration (in extract_from_section method, around lines 416-537):

After UTF-8 extraction block (around line 534), add UTF-16LE extraction logic:

Check if Encoding::Utf16Le is in config.encodings or config.enabled_encodings

Create Utf16ExtractionConfig using config.min_wide_length and config.utf16_min_confidence

Call utf16::extract_from_section() with section, data, config, noise filter config, and thresholds

Extend found_strings with the UTF-16LE results

Follow the same pattern as UTF-8 extraction: build filter context, apply noise filtering if enabled, filter by confidence threshold

Documentation Updates:

Update module-level documentation (lines 1-94) to mention UTF-16LE extraction support

Add UTF-16LE to the list of supported encodings in the module documentation

Integration Pattern:
The UTF-16LE extraction should be added after the UTF-8 extraction block and before the Ok(found_strings) return statement. It should reuse the same noise_filter_config and filter instances created for ASCII/UTF-8 extraction to maintain consistency and avoid redundant filter creation.

📄 src/lib.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update library-level documentation and re-exports to include UTF-16LE extraction support.

Changes Required:

Documentation Update (around lines 52-60):

Update the extraction module description to mention UTF-16LE support alongside ASCII and PE resources

Change the status indicator from "ASCII extraction and PE resources complete" to "ASCII, UTF-16LE extraction and PE resources complete"

Re-exports Update (around line 76):

Add Utf16ExtractionConfig to the re-exports from the extraction module

Update the re-export line to: pub use extraction::{AsciiExtractionConfig, Utf16ExtractionConfig, BasicExtractor, ExtractionConfig, StringExtractor};

Purpose:
These changes ensure that users of the library can easily access the new UTF-16LE extraction functionality through the top-level API without needing to navigate into the extraction submodule.

📄 tests/test_utf16_extraction.rs ^(NEW) 🔗

References

📄 tests/test_ascii_extraction.rs

📄 src/extraction/utf16.rs ^(NEW)

📄 src/types.rs

Create comprehensive unit tests for UTF-16LE string extraction, following the pattern established in tests/test_ascii_extraction.rs.

Test Coverage:

Basic Extraction Tests:

test_basic_utf16le_extraction: Extract simple UTF-16LE strings with null terminators

test_minimum_character_length: Verify minimum character length filtering (default 3 characters = 6 bytes)

test_custom_minimum_length: Test custom minimum length configuration

test_null_terminated_strings: Verify proper null termination detection (0x00 0x00)

Confidence Scoring Tests:

test_high_confidence_strings: Strings with >90% printable characters and null termination

test_medium_confidence_strings: Strings with 70-90% printable characters

test_low_confidence_strings: Strings with 50-70% printable characters

test_confidence_threshold_filtering: Verify strings below threshold are filtered out

Edge Cases:

test_empty_input: Empty data returns empty vector

test_no_valid_strings: Binary data with no valid UTF-16LE sequences

test_string_at_start: String at buffer start (offset 0)

test_string_at_end: String at buffer end without null terminator

test_odd_length_data: Handle odd-length data gracefully

test_partial_string_at_boundary: Partial UTF-16LE character at section boundary

test_interleaved_binary_data: UTF-16LE strings mixed with binary data

Section Integration Tests:

test_section_metadata_attachment: Verify section name, offset, and RVA are correctly populated

test_section_boundary_handling: Ensure extraction respects section boundaries

test_section_out_of_bounds: Handle sections beyond data bounds

test_different_section_types: Test extraction from various section types (.rdata, .data, .rsrc)

Real-World Scenarios:

test_windows_pe_versioninfo: Simulate VERSIONINFO resource strings (e.g., "Microsoft Corporation", "Copyright © 2024")

test_dotnet_string_literals: Simulate .NET assembly string literals

test_registry_paths: Test Windows registry path extraction

Noise Filtering Integration:

test_noise_filtering_enabled: Verify noise filtering reduces confidence for noisy strings

test_noise_filtering_disabled: Verify all strings extracted when filtering disabled

test_confidence_threshold_application: Test various confidence thresholds

Configuration Tests:

test_config_defaults: Verify default configuration values

test_config_customization: Test custom min_char_len, max_char_len, and min_confidence

test_max_length_filtering: Verify maximum length filtering works correctly

Test Data Helpers:

Create helper functions to generate UTF-16LE test data:

create_utf16le_string(text: &str) -> Vec<u8>: Convert ASCII string to UTF-16LE bytes

create_test_section(name: &str, offset: u64, size: u64, rva: Option<u64>) -> SectionInfo: Create test section metadata

create_utf16le_with_confidence(text: &str, noise_ratio: f32) -> Vec<u8>: Create UTF-16LE data with controlled noise for confidence testing

Assertions:

Verify encoding field is Encoding::Utf16Le

Verify source field is StringSource::SectionData

Verify confidence scores match expected ranges

Verify offset, length, section, and rva fields are correctly populated

Verify string text is correctly decoded to UTF-8

📄 tests/integration_extraction.rs ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

📄 tests/test_ascii_extraction.rs

Add integration tests for UTF-16LE extraction to verify end-to-end functionality with the BasicExtractor.

New Test Cases:

test_utf16le_extraction_with_basic_extractor:

Create a mock binary with UTF-16LE strings in a .rdata section

Use BasicExtractor with ExtractionConfig that includes Encoding::Utf16Le in enabled_encodings

Verify UTF-16LE strings are extracted with correct metadata

Verify confidence scores are applied

test_utf16le_encoding_filtering:

Test that UTF-16LE strings are only extracted when Encoding::Utf16Le is in enabled_encodings

Verify that disabling UTF-16LE encoding prevents extraction

test_utf16le_with_noise_filtering:

Create test data with both high-quality and noisy UTF-16LE strings

Enable noise filtering and verify low-confidence strings are filtered out

Disable noise filtering and verify all strings are extracted

test_utf16le_min_wide_length_config:

Test that min_wide_length configuration is respected

Verify strings shorter than min_wide_length are filtered out

test_utf16le_confidence_threshold:

Test various utf16_min_confidence threshold values

Verify strings below threshold are filtered out

Integration with Existing Tests:

Add UTF-16LE test cases to existing integration test functions if applicable

Ensure UTF-16LE extraction works alongside ASCII and UTF-8 extraction

Verify no conflicts or interference between different encoding extractors

Test Data:

Use realistic UTF-16LE strings from Windows PE binaries (version info, registry paths, error messages)

Include edge cases like very short strings, very long strings, and strings with special characters

📄 docs/src/string-extraction.md ^(MODIFY) 🔗

References

📄 src/extraction/utf16.rs ^(NEW)

📄 src/extraction/mod.rs ^(MODIFY)

Update the string extraction documentation to reflect the completed UTF-16LE implementation.

Changes Required:
UTF-16 Extraction Section (lines 212-283):

Update the section header to indicate UTF-16LE is now implemented (change from future tense to present tense)

Add a note that UTF-16LE extraction is complete and available

Update the implementation strategy section to reference the actual implementation in src/extraction/utf16.rs

Add usage examples showing how to use extract_utf16le_strings() and Utf16ExtractionConfig
Configuration Section (lines 404-418):
Update the ExtractionConfig documentation to include the new utf16_min_confidence field
Add example showing how to configure UTF-16LE extraction:
let mut config = ExtractionConfig::default();
config.min_wide_length = 3;
config.utf16_min_confidence = 0.7;
config.enabled_encodings.push(Encoding::Utf16Le);
Encoding Support Section (lines 11-87):

Update the encoding support list to show UTF-16LE as implemented (✅)

Add a brief description of UTF-16LE extraction capabilities

Examples Section:

Add practical examples of UTF-16LE extraction from Windows PE binaries:

Extracting VERSIONINFO strings

Extracting .NET string literals

Extracting registry paths

Include code snippets showing configuration and usage

Confidence Scoring Section (lines 276-282):

Expand the confidence scoring explanation for UTF-16LE:

Explain how printable character percentage is calculated

Describe the null termination bonus

Provide examples of high, medium, and low confidence strings
Documentation Style:

Maintain consistency with existing documentation style

Use clear, concise language

Include practical examples and use cases

Reference the actual implementation files for developers who want to dive deeper

Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module as utf16.rs
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if Utf16Le in enabled_encodings
    
    alt UTF-16LE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, config)
        Utf16Module->>Utf16Module: Extract section data slice
        Utf16Module->>Utf16Module: extract_utf16le_strings(data, config)
        
        loop For each even offset
            Utf16Module->>Utf16Module: Scan for printable UTF-16LE chars
            Utf16Module->>Utf16Module: Detect null termination (0x00 0x00)
            Utf16Module->>Utf16Module: Calculate confidence score
            
            alt Meets min_char_len & min_confidence
                Utf16Module->>Utf16Module: Decode to UTF-8 String
                Utf16Module->>FoundString: Create with Encoding::Utf16Le
            end
        end
        
        Utf16Module->>NoiseFilter: calculate_confidence(text, context)
        NoiseFilter-->>Utf16Module: confidence score
        
        alt confidence >= threshold
            Utf16Module->>Utf16Module: Adjust offsets, populate metadata
            Utf16Module-->>BasicExtractor: Vec<FoundString>
        end
        
        BasicExtractor->>BasicExtractor: Extend found_strings
    end
    
    BasicExtractor-->>User: All extracted strings

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-structured extraction framework with:

1. **Existing Infrastructure**: ASCII extractor (`src/extraction/ascii.rs`) serves as the reference implementation with comprehensive testing and noise filtering integration.

2. **Configuration Ready**: `ExtractionConfig` already has `min_wide_length` field (default: 3) for UTF-16 strings, and the `Encoding::Utf16Le` enum variant exists in `types.rs`.

3. **Noise Filtering**: The `CompositeNoiseFilter` in `src/extraction/filters.rs` operates on decoded UTF-8 text (the `text` field of `FoundString`), so it will work seamlessly with UTF-16LE strings after decoding.

4. **UTF-16LE Decoding**: The `decode_utf16le` function already exists in `src/extraction/pe_resources.rs` and can be reused or adapted for the new UTF-16LE extractor.

5. **Integration Pattern**: The `BasicExtractor::extract_from_section` method in `src/extraction/mod.rs` orchestrates extraction by calling specialized extractors (ASCII, UTF-8) and applying noise filtering with confidence thresholds.

6. **Testing Pattern**: Comprehensive unit tests in `tests/test_ascii_extraction.rs` provide a clear template for UTF-16LE tests.

### Approach

Create a new `src/extraction/utf16.rs` module implementing UTF-16LE string extraction with confidence scoring, following the established ASCII extractor pattern. The module will provide `extract_utf16le_strings()` for raw extraction and `extract_from_section()` for section-aware extraction with metadata population. Integration into `BasicExtractor::extract_from_section()` will check for `Encoding::Utf16Le` in `enabled_encodings` and apply the same noise filtering pipeline used for ASCII/UTF-8. Configuration will use the existing `min_wide_length` field and add `utf16_min_confidence` to `ExtractionConfig`. Comprehensive unit tests will cover basic extraction, confidence scoring, null termination, edge cases, and integration with section metadata.

### Reasoning

I explored the codebase structure by listing directories, reading key files including type definitions, extraction modules (ASCII, filters, PE resources), configuration structures, and test files. I examined the existing ASCII extractor pattern, noise filtering infrastructure, UTF-16LE decoding in PE resources, and the BasicExtractor integration point. I searched for UTF-16 references to understand current usage and verified the configuration fields and dependencies in Cargo.toml.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant BasicExtractor
    participant Utf16Module as utf16.rs
    participant NoiseFilter
    participant FoundString

    User->>BasicExtractor: extract_from_section(data, section, config)
    BasicExtractor->>BasicExtractor: Check if Utf16Le in enabled_encodings
    
    alt UTF-16LE enabled
        BasicExtractor->>Utf16Module: extract_from_section(section, data, config)
        Utf16Module->>Utf16Module: Extract section data slice
        Utf16Module->>Utf16Module: extract_utf16le_strings(data, config)
        
        loop For each even offset
            Utf16Module->>Utf16Module: Scan for printable UTF-16LE chars
            Utf16Module->>Utf16Module: Detect null termination (0x00 0x00)
            Utf16Module->>Utf16Module: Calculate confidence score
            
            alt Meets min_char_len & min_confidence
                Utf16Module->>Utf16Module: Decode to UTF-8 String
                Utf16Module->>FoundString: Create with Encoding::Utf16Le
            end
        end
        
        Utf16Module->>NoiseFilter: calculate_confidence(text, context)
        NoiseFilter-->>Utf16Module: confidence score
        
        alt confidence >= threshold
            Utf16Module->>Utf16Module: Adjust offsets, populate metadata
            Utf16Module-->>BasicExtractor: Vec<FoundString>
        end
        
        BasicExtractor->>BasicExtractor: Extend found_strings
    end
    
    BasicExtractor-->>User: All extracted strings

## Proposed File Changes

### src/extraction/utf16.rs(NEW)

References: 

- src/extraction/ascii.rs
- src/extraction/pe_resources.rs
- src/types.rs
- src/extraction/config.rs
- src/extraction/filters.rs

Create a new UTF-16LE string extraction module following the pattern established in `src/extraction/ascii.rs`.

**Module Structure:**

1. **Configuration Struct**: Define `Utf16ExtractionConfig` with fields:
   - `min_char_len: usize` (minimum character count, default: 3)
   - `max_char_len: Option<usize>` (maximum character count, default: None)
   - `min_confidence: f32` (minimum confidence threshold, default: 0.7)

2. **Helper Functions**:
   - `is_printable_utf16le_char(low: u8, high: u8) -> bool`: Check if a UTF-16LE character is printable (high byte is 0 and low byte is in 0x20-0x7E range, or common whitespace like tab/newline/carriage return)
   - `decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()>`: Convert UTF-16LE byte sequence to UTF-8 String using `u16::from_le_bytes` and `String::from_utf16`, similar to the implementation in `src/extraction/pe_resources.rs` but adapted for extraction use case
   - `calculate_confidence(data: &[u8], char_count: usize, has_null_terminator: bool) -> f32`: Calculate confidence score based on percentage of printable characters, proper null termination (bonus), and character count

3. **Core Extraction Function**: `extract_utf16le_strings(data: &[u8], config: &Utf16ExtractionConfig) -> Vec<FoundString>`:
   - Scan through data at even offsets (UTF-16LE requires 2-byte alignment)
   - For each position, accumulate printable UTF-16LE characters
   - Detect null termination (0x00 0x00 sequence)
   - Calculate confidence score for each candidate
   - Filter by minimum character length and confidence threshold
   - Convert to `FoundString` with `Encoding::Utf16Le`, proper offset, length (in bytes), and confidence score
   - Handle edge cases: partial strings at buffer boundaries, strings at start/end of data

4. **Section-Aware Extraction**: `extract_from_section(section: &SectionInfo, data: &[u8], config: &Utf16ExtractionConfig, noise_filter_config: Option<&NoiseFilterConfig>, noise_filtering_enabled: bool, min_confidence_threshold: f32) -> Vec<FoundString>`:
   - Extract section data slice with bounds checking (similar to `src/extraction/ascii.rs`)
   - Call `extract_utf16le_strings()` on section data
   - Build `FilterContext` from section metadata
   - Apply noise filtering if enabled using `CompositeNoiseFilter`
   - Adjust offsets to absolute positions (section.offset + relative_offset)
   - Populate section name and calculate RVA if available
   - Filter by confidence threshold

**Implementation Details:**
- Follow the algorithm pseudocode from the GitHub issue and `docs/src/string-extraction.md` (lines 238-274)
- Use the same error handling and bounds checking patterns as `src/extraction/ascii.rs`
- Ensure confidence scoring considers: printable character percentage, null termination presence, and string length
- High confidence (>0.9): >90% printable with proper null termination
- Medium confidence (0.7-0.9): >70% printable, reasonable length
- Low confidence (<0.7): >50% printable, may be coincidental

**Documentation:**
- Add comprehensive module-level documentation explaining UTF-16LE extraction, confidence scoring, and usage examples
- Document all public functions with examples
- Include notes on Windows PE binary string extraction use cases

### src/extraction/mod.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/ascii.rs
- src/types.rs

Integrate UTF-16LE extraction into the main extraction pipeline.

**Changes Required:**

1. **Module Declaration** (after line 110):
   - Add `pub mod utf16;` to declare the new UTF-16LE extraction module
   - Add public re-export: `pub use utf16::{extract_utf16le_strings, Utf16ExtractionConfig};` in the re-exports section (around line 110)

2. **ExtractionConfig Enhancement** (around lines 133-183):
   - Add new field `utf16_min_confidence: f32` with default value 0.7 in the `ExtractionConfig` struct (around line 159)
   - Update the `Default` implementation to include `utf16_min_confidence: 0.7` (around line 180)
   - Add validation in `ExtractionConfig::validate()` method to ensure `utf16_min_confidence` is between 0.0 and 1.0 (around line 205)

3. **BasicExtractor Integration** (in `extract_from_section` method, around lines 416-537):
   - After UTF-8 extraction block (around line 534), add UTF-16LE extraction logic:
     - Check if `Encoding::Utf16Le` is in `config.encodings` or `config.enabled_encodings`
     - Create `Utf16ExtractionConfig` using `config.min_wide_length` and `config.utf16_min_confidence`
     - Call `utf16::extract_from_section()` with section, data, config, noise filter config, and thresholds
     - Extend `found_strings` with the UTF-16LE results
   - Follow the same pattern as UTF-8 extraction: build filter context, apply noise filtering if enabled, filter by confidence threshold

4. **Documentation Updates**:
   - Update module-level documentation (lines 1-94) to mention UTF-16LE extraction support
   - Add UTF-16LE to the list of supported encodings in the module documentation

**Integration Pattern:**
The UTF-16LE extraction should be added after the UTF-8 extraction block and before the `Ok(found_strings)` return statement. It should reuse the same `noise_filter_config` and `filter` instances created for ASCII/UTF-8 extraction to maintain consistency and avoid redundant filter creation.

### src/lib.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update library-level documentation and re-exports to include UTF-16LE extraction support.

**Changes Required:**

1. **Documentation Update** (around lines 52-60):
   - Update the extraction module description to mention UTF-16LE support alongside ASCII and PE resources
   - Change the status indicator from "ASCII extraction and PE resources complete" to "ASCII, UTF-16LE extraction and PE resources complete"

2. **Re-exports Update** (around line 76):
   - Add `Utf16ExtractionConfig` to the re-exports from the extraction module
   - Update the re-export line to: `pub use extraction::{AsciiExtractionConfig, Utf16ExtractionConfig, BasicExtractor, ExtractionConfig, StringExtractor};`

**Purpose:**
These changes ensure that users of the library can easily access the new UTF-16LE extraction functionality through the top-level API without needing to navigate into the extraction submodule.

### tests/test_utf16_extraction.rs(NEW)

References: 

- tests/test_ascii_extraction.rs
- src/extraction/utf16.rs(NEW)
- src/types.rs

Create comprehensive unit tests for UTF-16LE string extraction, following the pattern established in `tests/test_ascii_extraction.rs`.

**Test Coverage:**

1. **Basic Extraction Tests**:
   - `test_basic_utf16le_extraction`: Extract simple UTF-16LE strings with null terminators
   - `test_minimum_character_length`: Verify minimum character length filtering (default 3 characters = 6 bytes)
   - `test_custom_minimum_length`: Test custom minimum length configuration
   - `test_null_terminated_strings`: Verify proper null termination detection (0x00 0x00)

2. **Confidence Scoring Tests**:
   - `test_high_confidence_strings`: Strings with >90% printable characters and null termination
   - `test_medium_confidence_strings`: Strings with 70-90% printable characters
   - `test_low_confidence_strings`: Strings with 50-70% printable characters
   - `test_confidence_threshold_filtering`: Verify strings below threshold are filtered out

3. **Edge Cases**:
   - `test_empty_input`: Empty data returns empty vector
   - `test_no_valid_strings`: Binary data with no valid UTF-16LE sequences
   - `test_string_at_start`: String at buffer start (offset 0)
   - `test_string_at_end`: String at buffer end without null terminator
   - `test_odd_length_data`: Handle odd-length data gracefully
   - `test_partial_string_at_boundary`: Partial UTF-16LE character at section boundary
   - `test_interleaved_binary_data`: UTF-16LE strings mixed with binary data

4. **Section Integration Tests**:
   - `test_section_metadata_attachment`: Verify section name, offset, and RVA are correctly populated
   - `test_section_boundary_handling`: Ensure extraction respects section boundaries
   - `test_section_out_of_bounds`: Handle sections beyond data bounds
   - `test_different_section_types`: Test extraction from various section types (.rdata, .data, .rsrc)

5. **Real-World Scenarios**:
   - `test_windows_pe_versioninfo`: Simulate VERSIONINFO resource strings (e.g., "Microsoft Corporation", "Copyright © 2024")
   - `test_dotnet_string_literals`: Simulate .NET assembly string literals
   - `test_registry_paths`: Test Windows registry path extraction

6. **Noise Filtering Integration**:
   - `test_noise_filtering_enabled`: Verify noise filtering reduces confidence for noisy strings
   - `test_noise_filtering_disabled`: Verify all strings extracted when filtering disabled
   - `test_confidence_threshold_application`: Test various confidence thresholds

7. **Configuration Tests**:
   - `test_config_defaults`: Verify default configuration values
   - `test_config_customization`: Test custom min_char_len, max_char_len, and min_confidence
   - `test_max_length_filtering`: Verify maximum length filtering works correctly

**Test Data Helpers**:
- Create helper functions to generate UTF-16LE test data:
  - `create_utf16le_string(text: &str) -> Vec<u8>`: Convert ASCII string to UTF-16LE bytes
  - `create_test_section(name: &str, offset: u64, size: u64, rva: Option<u64>) -> SectionInfo`: Create test section metadata
  - `create_utf16le_with_confidence(text: &str, noise_ratio: f32) -> Vec<u8>`: Create UTF-16LE data with controlled noise for confidence testing

**Assertions:**
- Verify `encoding` field is `Encoding::Utf16Le`
- Verify `source` field is `StringSource::SectionData`
- Verify `confidence` scores match expected ranges
- Verify `offset`, `length`, `section`, and `rva` fields are correctly populated
- Verify string text is correctly decoded to UTF-8

### tests/integration_extraction.rs(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)
- tests/test_ascii_extraction.rs

Add integration tests for UTF-16LE extraction to verify end-to-end functionality with the `BasicExtractor`.

**New Test Cases:**

1. **test_utf16le_extraction_with_basic_extractor**:
   - Create a mock binary with UTF-16LE strings in a .rdata section
   - Use `BasicExtractor` with `ExtractionConfig` that includes `Encoding::Utf16Le` in `enabled_encodings`
   - Verify UTF-16LE strings are extracted with correct metadata
   - Verify confidence scores are applied

2. **test_utf16le_encoding_filtering**:
   - Test that UTF-16LE strings are only extracted when `Encoding::Utf16Le` is in `enabled_encodings`
   - Verify that disabling UTF-16LE encoding prevents extraction

3. **test_utf16le_with_noise_filtering**:
   - Create test data with both high-quality and noisy UTF-16LE strings
   - Enable noise filtering and verify low-confidence strings are filtered out
   - Disable noise filtering and verify all strings are extracted

4. **test_utf16le_min_wide_length_config**:
   - Test that `min_wide_length` configuration is respected
   - Verify strings shorter than `min_wide_length` are filtered out

5. **test_utf16le_confidence_threshold**:
   - Test various `utf16_min_confidence` threshold values
   - Verify strings below threshold are filtered out

**Integration with Existing Tests:**
- Add UTF-16LE test cases to existing integration test functions if applicable
- Ensure UTF-16LE extraction works alongside ASCII and UTF-8 extraction
- Verify no conflicts or interference between different encoding extractors

**Test Data:**
- Use realistic UTF-16LE strings from Windows PE binaries (version info, registry paths, error messages)
- Include edge cases like very short strings, very long strings, and strings with special characters

### docs/src/string-extraction.md(MODIFY)

References: 

- src/extraction/utf16.rs(NEW)
- src/extraction/mod.rs(MODIFY)

Update the string extraction documentation to reflect the completed UTF-16LE implementation.

**Changes Required:**

1. **UTF-16 Extraction Section** (lines 212-283):
   - Update the section header to indicate UTF-16LE is now implemented (change from future tense to present tense)
   - Add a note that UTF-16LE extraction is complete and available
   - Update the implementation strategy section to reference the actual implementation in `src/extraction/utf16.rs`
   - Add usage examples showing how to use `extract_utf16le_strings()` and `Utf16ExtractionConfig`

2. **Configuration Section** (lines 404-418):
   - Update the `ExtractionConfig` documentation to include the new `utf16_min_confidence` field
   - Add example showing how to configure UTF-16LE extraction:
     ```rust
     let mut config = ExtractionConfig::default();
     config.min_wide_length = 3;
     config.utf16_min_confidence = 0.7;
     config.enabled_encodings.push(Encoding::Utf16Le);
     ```

3. **Encoding Support Section** (lines 11-87):
   - Update the encoding support list to show UTF-16LE as implemented (✅)
   - Add a brief description of UTF-16LE extraction capabilities

4. **Examples Section**:
   - Add practical examples of UTF-16LE extraction from Windows PE binaries:
     - Extracting VERSIONINFO strings
     - Extracting .NET string literals
     - Extracting registry paths
   - Include code snippets showing configuration and usage

5. **Confidence Scoring Section** (lines 276-282):
   - Expand the confidence scoring explanation for UTF-16LE:
     - Explain how printable character percentage is calculated
     - Describe the null termination bonus
     - Provide examples of high, medium, and low confidence strings

**Documentation Style:**
- Maintain consistency with existing documentation style
- Use clear, concise language
- Include practical examples and use cases
- Reference the actual implementation files for developers who want to dive deeper

Developer Humor

🎯 UTF-16LE strings, once hidden in plain sight,\n> Now extracted with confidence, shining bright!\n> From Windows PEs to .NET's embrace,\n> Wide characters find their rightful place.\n> 🔍 Two bytes per char, but worth the space! 💾

Execution Information

Branch: main
Commit: c50e9b6

💡 Tips

Supported Commands (Inside Comments)

Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Implement UTF-16LE String Extraction with Confidence Scoring and Minimum Length Filtering #11

Description

Summary

Context

Technical Background

UTF-16LE Detection Heuristics

Confidence Scoring

Requirements

Core Functionality

Configuration

Integration

Testing

Proposed Implementation

File Structure

Algorithm Pseudocode

Acceptance Criteria

Example Use Cases

VERSIONINFO Extraction from PE

.NET String Literal Detection

Dependencies

References

Task Metadata

Activity

unclesp1d3r commented on Sep 30, 2025

coderabbitai commented on Sep 30, 2025

coderabbitai commented on Sep 30, 2025

🎯 Title Enhancement

📝 Body Enhancements

Added Comprehensive Sections:

Key Technical Details Added:

🏷️ Metadata Status

unclesp1d3r commented on Nov 13, 2025

traycerai commented on Nov 13, 2025

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

traycerai commented on Nov 13, 2025

Plan

Observations

Approach

Reasoning

Proposed File Changes

Diagram

Import In IDE

Developer Humor

Supported Commands (Inside Comments)

Supported Commands (Inside Description)

Community

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions