Data Flow

Data Flow and Processing Pipeline

File Upload Process

1. User uploads file(s) through the frontend interface (GNSS and/or IMU files)
2. Files are temporarily stored in the backend's upload directory
3. Files are uploaded to Azure Blob Storage in the 'uploads' container
4. A processing job is created in the Redis-backed Bull queue
5. Job ID is returned to the frontend for status tracking

Processing Pipeline

1. Worker picks up the job from the queue
2. File format is detected based on extension and content analysis
3. AI-assisted conversion is attempted first using Azure OpenAI
4. If AI conversion fails, fallback to standard format-specific conversion
5. Converted data is validated and standardized
6. If both GNSS and IMU files are provided, fusion is performed (not fully implemented)
7. Results are stored in Azure Blob Storage in the 'processed' container
8. Job status is updated with result URLs

Conversion Process Flow

Format-Specific Processing

RINEX Files

1. Parse header section for metadata
2. Extract observation records
3. Convert to standardized format
4. Generate JSONL output

NMEA Files

1. Split into NMEA sentences
2. Parse each sentence by type
3. Extract coordinates and metadata
4. Normalize timestamps
5. Generate JSONL output

UBX Binary Files

1. Parse binary structure
2. Extract message payloads
3. Decode based on message type
4. Combine related messages
5. Generate JSONL output

Fallback Mechanism

When both AI and format-specific processing fail:

1. Basic line-by-line conversion
2. Field extraction using regular expressions
3. Type detection for values
4. Basic schema creation