Import

🎯 Overview

The Import Page serves as the central hub for uploading and processing various types of data files into the system. This comprehensive interface allows users to import project data, registrations, and other structured information through a streamlined file upload and job processing system. The page provides configurable import categories, real-time processing status monitoring, and detailed job management capabilities.

The Import Module operates with specialized components for different data types:

📊 Registration Imports: User registration and enrollment data with template downloads
📁 Configurable Import Categories: Flexible import types based on project configuration
🔄 Job Processing System: Background processing with progress tracking and error handling

🔐 Access Requirements

👥 Required Roles & Permissions

View Imports: imports.view permission
Upload Files: imports.insert permission
Cancel Jobs: imports.cancel permission
Remove Files/Jobs: imports.remove permission
Registration Imports: registrationImports.insert permission

🌐 Permission Scope

Project-Based: Access limited to imports within user's assigned projects
Role-Based Operations: Different actions available based on user permissions
File-Level Security: Individual file access validation

💻 Import Page Interface Layout

🎛️ Main Import Overview

┌──────────────────────────────────────────────────────────────────────────────┐
│                               Import Overview                                │
├──────────────────────────────────────────────────────────────────────────────┤
│ Registrations                                                                │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Registrations                                            (i) Remarks     │ │
│ │ [Browse Files]   [Download Template]                                     │ │
│ │                                                                          │ │
│ │ Upload History:                                                          │ │
│ │ ├── [✓] registrations_Nov3.xlsx (Completed)                              │ │
│ │ │     Date: Nov 3, 2025 14:30   [Download] [Remove]                      │ │
│ │ ├── [↻] registrations_Nov2.xlsx (Running 75%)                            │ │
│ │ │     Progress: ███████▒▒  [Cancel] [Details]                            │ │
│ │ └── [✗] registrations_Nov1.xlsx (Failed)                                 │ │
│ │       Reason: Invalid data format  [Remove] [Details]                    │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
├──────────────────────────────────────────────────────────────────────────────┤
│ Data Files                                                                   │
│ ┌───────────────┬───────────────┬───────────────┐                            │
│ │ Product Data  │ Zones         │ Users         │                            │
│ │ [Browse]      │ [Browse]      │ [Browse]      │                            │
│ │ Status: Ready │ Status: 75%   │ Status: Failed│                            │
│ │ [Download]    │ Progress: ███████▒▒  [Cancel] │ [Remove] [Details]         │
│ └───────────────┴───────────────┴───────────────┘                            │
├──────────────────────────────────────────────────────────────────────────────┤
│ Configuration Files                                                          │
│ ┌───────────────────────┬───────────────────────┐                            │
│ │ Settings              │ Rules                 │                            │
│ │ [Browse Files]        │ [Browse Files]        │                            │
│ │ Status: Processing    │ Status: Completed     │                            │
│ │ [Cancel]              │ [Download]            │                            │
│ └───────────────────────┴───────────────────────┘                            │
└──────────────────────────────────────────────────────────────────────────────┘

📁 File Upload Interface

File Upload Component
┌──────────────────────────────────────────────────────────────────────────────┐
│ File Category: Product Data                                       (i) Info   │
├──────────────────────────────────────────────────────────────────────────────┤
│ Upload Zone:                                                                 │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │                     [Browse Files]                                       │ │
│ │           Drag & drop files here or click to browse                      │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
├──────────────────────────────────────────────────────────────────────────────┤
│ Processing Progress: 45%                                                     │
│ █████▒▒▒▒▒  [Cancel]                                                         │
└──────────────────────────────────────────────────────────────────────────────┘

📥 Registration Imports Component

The Registration Imports section provides specialized functionality for importing user registration data with template support and validation.

⭐ Registration Import Features

Template Download: Pre-configured Excel templates for registration data
Validation Support: Built-in data validation and error reporting
Bulk Registration: Process multiple user registrations simultaneously
Progress Tracking: Real-time upload and processing progress monitoring

📋 Registration Import Workflow

Download Template: Get the standardized registration template
Fill Data: Complete registration information in the template
Upload File: Browse and select the completed registration file
Monitor Progress: Track processing status and validation results
Review Results: Check for errors and successful registrations

📁 Configurable Import Categories

The system supports flexible import categories based on project configuration, allowing for different types of data imports.

⭐ Import Category Features

Dynamic Configuration: Categories defined by project ImportConfig settings
File Type Validation: Automatic validation of file formats and naming conventions
Grouped Organization: Logical grouping of related import types
Custom Remarks: Contextual information and instructions for each category

📊 Common Import Categories

📦 Product Data: Product catalogs, descriptions, and specifications
🏢 Zone Information: Zone configurations and layout data
👥 User Data: User accounts and profile information
⚙️ Configuration Files: System settings and rule configurations
📋 Master Data: Reference data and lookup tables

⭐ Core Import System Features

🔄 Job Processing Engine

Background Processing: Non-blocking file processing with job queue management
Progress Monitoring: Real-time progress updates with percentage completion
Error Handling: Comprehensive error reporting with detailed failure messages
Job Management: Cancel, remove, and restart capabilities for import jobs

📁 File Management System

Multi-format Support: Excel, CSV, XML, and other structured file formats
File Validation: Pre-processing validation of file structure and content
Version Control: Track multiple versions of imported files
Download Access: Download original files and processing logs

🔍 Status Monitoring

Real-time Updates: Live status updates during file processing
Detailed Logging: Comprehensive logs for troubleshooting and auditing
Warning Management: Handle warnings and partial import scenarios
History Tracking: Complete audit trail of all import operations

📘 Import Page Usage Guide

🚀 Getting Started with Imports

Navigate to Import page via main navigation menu
Review Categories: Check available import categories for your project
Select Import Type: Choose the appropriate category for your data
Prepare Files: Ensure files meet format and naming requirements
Upload Files: Use browse button or drag-and-drop functionality

📋 Registration Import Process

Download Template:

📥 Click "Download Template" in Registrations section
💾 Save the Excel template to your computer
📝 Review template structure and required fields

Prepare Registration Data:

📊 Open downloaded template in Excel or compatible software
📝 Fill in registration information for each user
✅ Validate data format and required fields
💾 Save completed template with descriptive filename

Upload Registration File:

📤 Click "Browse Files" in Registrations section
📁 Select your completed registration file
🔄 Monitor upload progress and processing status
✅ Review processing results and any error messages

📁 Standard File Import Process

Select Import Category:

📊 Choose appropriate category (Product Data, Zones, etc.)
ℹ️ Review any remarks or instructions for the category
📋 Check file format requirements and naming conventions

Upload Files:

📤 Click "Browse Files" for desired category
📁 Select file(s) from your computer
🔄 Monitor upload progress with progress bar
⏸️ Use Cancel button if needed to stop upload

Monitor Processing:

📊 Watch real-time progress percentage
📋 Review processing status (Running, Completed, Failed)
🔍 Click "Details" to view processing logs
⚠️ Check for warnings or validation errors

🛠️ Managing Import Jobs

Job Status Monitoring:
- ⏳ Running: Job is actively processing (shows progress %)
- ✅ Completed: Job finished successfully (may have warnings)
- ❌ Failed: Job encountered errors and stopped
- 🚫 Cancelled: Job was manually cancelled by user

Job Actions:

🚫 Cancel: Stop running jobs before completion
🗑️ Remove: Delete completed/failed jobs and files
🔍 Details: View comprehensive processing logs
📥 Download: Access original uploaded files

Error Resolution:

📋 Review error messages in failed job details
📝 Correct data issues in original files
🔄 Re-upload corrected files
✅ Verify successful processing

📊 Understanding Import Status Indicators

📤 Uploaded: File successfully uploaded, waiting for processing
🔄 Processing: File is being validated and imported
✅ Completed: Import finished successfully
⚠️ Completed with Warnings: Import finished but with non-critical issues
❌ Failed: Import failed due to validation or processing errors
🚫 Cancelled: Import was manually stopped by user

⚠️ Import Page Troubleshooting

🚫 Common Import Issues

❌ "Filename does not match regular expression" Error

✅ Check Naming Convention: Follow required filename pattern for category
✅ Remove Special Characters: Use only allowed characters in filename
✅ File Extension: Ensure correct file extension (.xlsx, .csv, etc.)
✅ Review Requirements: Check category remarks for specific naming rules

❌ "File type is not allowed" Error

✅ Format Validation: Use only supported file formats for category
✅ File Extension: Verify file has correct extension
✅ Content Type: Ensure file content matches declared format
✅ Template Usage: Use provided templates when available

❌ "Import failed" with Data Errors

✅ Data Validation: Check for missing required fields
✅ Format Consistency: Ensure data follows expected format patterns
✅ Template Compliance: Verify data matches template structure
✅ Review Logs: Check detailed error logs for specific issues

❌ "Permission denied" Error

✅ Role Verification: Confirm adequate permissions for import operations
✅ Project Access: Verify membership in target project
✅ Category Access: Check if user has access to specific import category
✅ Admin Contact: Request appropriate permissions from administrator

❌ "File linked to job" Error

✅ Job Completion: Wait for linked job to complete before removing file
✅ Cancel Job: Cancel running job before attempting file removal
✅ Job Status: Check job status and resolve any processing issues

📱 Mobile/Responsive Import Usage

File Selection: Touch-optimized file browser interface
Progress Monitoring: Mobile-friendly progress indicators
Job Management: Streamlined job controls for smaller screens
Upload Interface: Responsive file upload zones

🔄 Import Performance Optimization

File Size Management: Break large files into smaller chunks when possible
Peak Hours: Avoid large imports during high system usage
Format Selection: Use efficient file formats (CSV vs Excel for large datasets)
Batch Processing: Process multiple small files rather than one large file

🛠️ Import System Technical Details

🏗️ Import Architecture

// Import System Structure
{
  importOverview: {
    categories: [
      {
        ID: "productData",
        FileDescription: "Product Data Import",
        GroupName: "Data Files",
        Remarks: "CSV format required",
        IsHidden: false
      }
    ],
    registrationConfig: {
      templateGeneration: true,
      validationRules: ["email", "phone", "required"]
    }
  },
  fileProcessing: {
    uploadHandler: "GridFS",
    jobQueue: "background",
    validationEngine: "configurable"
  }
}

📁 Import File Structure

// Import File Record
{
  _id: "file123",
  name: "product_data_20241103.xlsx",
  size: 1048576,
  type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
  uploaded_at: new Date(),
  meta: {
    projectId: "project456",
    category: "productData",
    uploadedBy: "user789",
    originalName: "product_data_20241103.xlsx"
  }
}

🔄 Import Job Structure

// Import Job Processing
{
  _id: "job123",
  type: "import",
  status: "running", // waiting, running, completed, failed, cancelled
  progress: {
    percent: 75,
    processed: 750,
    total: 1000
  },
  data: {
    fileId: "file123",
    category: "productData",
    meta: {
      projectId: "project456",
      userId: "user789"
    },
    validation: {
      passed: 750,
      failed: 0,
      warnings: 25
    }
  },
  log: [
    { level: "info", message: "Processing started", timestamp: new Date() },
    { level: "warn", message: "Missing optional field", timestamp: new Date() }
  ],
  failures: [
    { value: "Row 15: Invalid email format", timestamp: new Date() }
  ]
}

🔐 Import Security Features

File Type Validation: Strict file format and content validation
Project Isolation: Import files isolated by project membership
Role-based Access: Hierarchical permission system for import operations
Audit Trail: Complete logging of all import activities
Input Sanitization: Comprehensive data sanitization during processing

📈 Import Performance Optimizations

Stream Processing: Large files processed in chunks to manage memory
Background Jobs: Non-blocking import processing with job queue
Validation Caching: Cache validation rules to improve processing speed
Database Optimization: Efficient bulk insert operations for large datasets
Progress Streaming: Real-time progress updates via reactive subscriptions

🔄 Integration Points

File Storage: Integration with GridFS or external storage systems
User Management: User validation and role checking during registration imports
Project Data: Direct integration with project-specific data models
Notification System: Import completion and error notifications
Configuration Management: Dynamic import categories based on project settings

📊 Import Validation Engine

// Validation Rule Engine
const validateImportData = (data, category, rules) => {
  const results = {
    passed: 0,
    failed: 0,
    warnings: 0,
    errors: []
  };
  
  data.forEach((row, index) => {
    try {
      const validation = applyValidationRules(row, rules);
      if (validation.isValid) {
        results.passed++;
      } else {
        results.failed++;
        results.errors.push({
          row: index + 1,
          message: validation.error
        });
      }
    } catch (error) {
      results.failed++;
      results.errors.push({
        row: index + 1,
        message: error.message
      });
    }
  });
  
  return results;
};

This documentation covers the comprehensive Import system with file upload, processing, and management capabilities. The system supports both standard data imports and specialized registration imports with template support. For advanced import configuration, custom validation rules, or troubleshooting complex import issues, consult with your system administrator.

🎯 Overview​

🔐 Access Requirements​

👥 Required Roles & Permissions​

🌐 Permission Scope​

💻 Import Page Interface Layout​

🎛️ Main Import Overview​

📁 File Upload Interface​

📥 Registration Imports Component​

⭐ Registration Import Features​

📋 Registration Import Workflow​

📁 Configurable Import Categories​

⭐ Import Category Features​

📊 Common Import Categories​

⭐ Core Import System Features​

🔄 Job Processing Engine​

📁 File Management System​

🔍 Status Monitoring​

📘 Import Page Usage Guide​

🚀 Getting Started with Imports​

📋 Registration Import Process​

📁 Standard File Import Process​

🛠️ Managing Import Jobs​

📊 Understanding Import Status Indicators​

⚠️ Import Page Troubleshooting​

🚫 Common Import Issues​

📱 Mobile/Responsive Import Usage​

🔄 Import Performance Optimization​

🛠️ Import System Technical Details​

🏗️ Import Architecture​

📁 Import File Structure​

🔄 Import Job Structure​

🔐 Import Security Features​

📈 Import Performance Optimizations​

🔄 Integration Points​

📊 Import Validation Engine​