Filesystem Trigger (Filesystem Change Watcher)

The Filesystem Trigger monitors directories for file changes and automatically starts workflows when files are modified or created.

What does this integration do?

The Filesystem Trigger monitors a specific directory in the filesystem and reacts to file changes. When a file is modified, it automatically reads the file content and starts the linked workflow with this content.

Typical Use Cases:

• Document Processing: Automatic processing of new documents in an upload folder
• Data Import: Processing CSV/JSON files as soon as they arrive
• Log Monitoring: Monitor log files for changes
• Backup Monitoring: Processing backup files after creation
• File Synchronization: Automatic further processing of synchronized files
• ETL Processes: Triggering data processing workflows

User Configuration

Monitoring Configuration

Path

• Purpose: Directory path in the filesystem to be monitored
• Format: Absolute or relative path
• Examples:
  • /home/workflows/incoming
  • C:\\Upload\\Documents
  • ./data/incoming
  • /var/uploads/

File-pattern to include (Optional)

• Purpose: Glob patterns to filter which files should be monitored
• Format: Semicolon-separated glob patterns
• Leave empty: All files will be monitored
• Examples:
  • *.pdf (only PDF files)
  • *.csv;*.json (CSV and JSON files)
  • report_*.xlsx (Excel files with specific naming scheme)
  • data_*.txt;log_*.log (multiple patterns)

File-pattern to exclude (Optional)

• Purpose: Glob patterns to exclude specific files
• Format: Semicolon-separated glob patterns
• Examples:
  • *.tmp (exclude temporary files)
  • .DS_Store;Thumbs.db (exclude system files)
  • backup_*;cache_* (exclude backup and cache files)

Data Output

Result (File) - Variable for File Content

• Purpose: Name of the variable that stores the content of the changed file
• Content: Complete text content of the file
• Example Variable: fileContent
• Encoding: UTF-8

How it Works

Monitoring Mechanism

File Watcher:

• Uses Node.js fs.watch() for efficient file monitoring
• Only reacts to change events (file modification)
• Ignores other events like rename or delete

Filter Processing:

1. Include Filter: If set, only matching files are processed
2. Exclude Filter: Matching files are excluded
3. Combined: Include filter is applied first, then exclude filter

File Processing:

• File content is read as UTF-8 text
• Complete content is stored in the configured variable
• Faulty files are logged but don't block further processing

Workflow Integration

Data Format

File Content (Result Variable):

# Example CSV file
Name,Email,Age
John Doe,john@example.com,30
Jane Smith,jane@example.com,25

JSON File:

{
  "orders": [
    { "id": 123, "customer": "John", "amount": 99.99 },
    { "id": 124, "customer": "Jane", "amount": 149.5 }
  ]
}

Usage in Subsequent Steps

Process File Content:

Variable: fileContent

In LLM Action:
"Analyze the following CSV data and create a summary: {{fileContent}}"

In Script Action:
const content = memory.load('fileContent');
const data = JSON.parse(content);

Structured Data Processing:

In Script Action:
const csvData = memory.load('fileContent');
const lines = csvData.split('\n');
const headers = lines[0].split(',');
// Further CSV processing...

Practical Examples

CSV Data Import

Configuration:

• Path: /data/imports/
• File-pattern to include: *.csv
• File-pattern to exclude: temp_*;backup_*
• Result: csvData

Usage: Automatic processing of CSV imports, data validation, database import

Document Processing

Configuration:

• Path: /documents/incoming/
• File-pattern to include: *.pdf;*.docx;*.txt
• Result: documentContent

Usage: OCR processing, document classification, archiving

Log Monitoring

Configuration:

• Path: /var/log/application/
• File-pattern to include: app_*.log
• File-pattern to exclude: debug_*;temp_*
• Result: logContent

Usage: Real-time log analysis, error detection, alert generation

Data Processing Pipeline

Configuration:

• Path: ./data/processing/
• File-pattern to include: data_*.json;report_*.xml
• Result: processingData

Usage: ETL processes, data validation, report generation

Technical Details

Schema Configuration

configSchema: {
  path: {
    name: 'Path',
    description: 'Path on file system to listen to',
    schema: z.string(),
  },
  includeFilter: {
    name: 'File-pattern to include',
    description: 'Semicolon separated glob patterns (optional)',
    schema: z.string().optional(),
  },
  excludeFilter: {
    name: 'File-pattern to exclude',
    description: 'Semicolon separated glob patterns (optional)',
    schema: z.string().optional(),
  },
  outputName: {
    name: 'Result (File)',
    reference: 'memory-out',
    schema: z.string(),
  },
}

Internal Implementation

File Monitoring:

• Uses fs.watch() for platform-specific, efficient monitoring
• Event-based architecture without polling
• Automatic cleanup on stop

Pattern Matching:

• Uses minimatch library for glob pattern support
• Flexible combinations of include and exclude filters
• Case-sensitive pattern matching

File Access:

• Synchronous reading for consistency
• UTF-8 encoding by default
• Robust error handling for access problems

Best Practices

Directory Management

Path Structuring:

• Use dedicated directories for different file types
• Implement clear naming conventions
• Use subdirectories for organization

Permissions:

• Ensure the workflow service has read permissions
• Monitor write permissions for external systems
• Implement security validation for file paths

Performance

Filter Optimization:

• Use specific include filters for performance improvement
• Exclude system and temporary files
• Limit directory sizes for better performance

File Sizes:

• Consider memory limits when reading large files
• Implement size validation in subsequent steps
• Use streaming for very large files

Robustness

Error Handling:

• Implement retry logic for temporarily locked files
• Validate file contents in subsequent workflow steps
• Handle different file encodings

Monitoring:

• Monitor file processing times
• Track error rates for file access
• Implement alerts for critical directories

Security

File Access:

• Validate file paths to prevent path traversal
• Implement size limitations
• Check file types before processing

Content Validation:

• Scan files for malicious content
• Implement data validation
• Use sandboxing for unknown file formats

---

The Filesystem Trigger enables seamless integration of filesystem events into automated workflows and provides flexible, efficient monitoring of file changes.