Skip to main content

Refresh Knowledge

Learn how to maintain, update, and optimize your knowledge bases to ensure they continue providing accurate and current information for your AI workflows.

Overview​

Knowledge maintenance involves:

  • Content Updates: Adding new and updated documents
  • Index Refresh: Rebuilding search indexes for optimal performance
  • Quality Assurance: Monitoring and improving content quality
  • Performance Optimization: Maintaining fast search response times
  • Content Cleanup: Removing outdated or irrelevant information

When to Refresh Knowledge​

Automatic Refresh Triggers​

File System Sources

  • New Files: Automatically detected and processed
  • Modified Files: Updated when source files change
  • Deleted Files: Removed from index when source files are deleted
  • Directory Changes: Monitors entire directory structures

Scheduled Refresh

  • Daily Updates: For frequently changing content
  • Weekly Refresh: For regularly updated documentation
  • Monthly Rebuild: For comprehensive optimization
  • Custom Schedules: Based on your content update patterns

Manual Refresh Scenarios​

Content Quality Issues

  • Poor search results indicate content problems
  • Outdated information needs to be updated
  • New document formats or structures require reprocessing
  • Index corruption or performance degradation

Structural Changes

  • Source directory reorganization
  • Web site structure changes
  • Document format updates
  • New content types added

Performance Optimization

  • Slow search response times
  • Growing content volumes
  • Index fragmentation
  • Resource usage optimization

Manual Refresh Process​

Refreshing Individual Knowledge Bases​

Step 1: Navigate to Knowledge

  1. Go to "Knowledge" in the main navigation
  2. Select the knowledge base you want to refresh
  3. Click on the knowledge base name to open it

Step 2: Initiate Refresh

  1. Go to the "Edit" tab
  2. Click "Refresh" button
  3. Confirm the refresh operation
  4. Monitor progress in the Jobs section

Step 3: Monitor Progress

  • Processing Status: Track refresh job progress
  • Error Monitoring: Watch for processing errors
  • Performance Impact: Monitor system resource usage
  • Completion Verification: Confirm refresh completes successfully

Bulk Refresh Operations​

Multiple Knowledge Bases

  1. Navigate to administration section
  2. Select "Knowledge Management" or similar option
  3. Choose multiple knowledge bases
  4. Initiate bulk refresh operation

Workspace-Wide Refresh

  • Refresh all knowledge bases in a workspace
  • Useful after system updates or optimizations
  • Can be scheduled during maintenance windows
  • Requires appropriate administrative permissions

Refresh Types​

Incremental Refresh​

What it Does

  • Processes only new or changed content
  • Faster than full refresh
  • Minimal impact on system resources
  • Preserves existing index structure

When to Use

  • Regular maintenance refreshes
  • After adding new documents
  • When source content has minor updates
  • For performance-sensitive environments

How it Works

  1. Change Detection: Identifies new or modified content
  2. Selective Processing: Only processes changed items
  3. Index Updates: Updates search index incrementally
  4. Optimization: Maintains index performance

Full Refresh​

What it Does

  • Reprocesses all content from scratch
  • Rebuilds search index completely
  • Optimizes index structure and performance
  • Ensures consistency and accuracy

When to Use

  • After major content changes
  • When experiencing performance issues
  • For periodic optimization
  • After system updates or configuration changes

How it Works

  1. Content Reprocessing: Re-extracts text from all sources
  2. Index Rebuild: Creates new search index
  3. Optimization: Optimizes index structure
  4. Validation: Verifies refresh completion

Selective Refresh​

Content-Specific

  • Refresh specific document types or sources
  • Update particular directories or URL patterns
  • Process specific file formats only
  • Focus on high-priority content

Source-Specific

  • Refresh individual source configurations
  • Update specific file system paths
  • Re-crawl specific web URLs
  • Process specific upload buckets

Monitoring Refresh Operations​

Real-Time Monitoring​

Refresh Jobs

  • Status Tracking: Monitor refresh job progress
  • Step Details: See which processing steps are running
  • Error Alerts: Immediate notification of issues
  • Resource Usage: Track CPU, memory, and storage usage

Progress Indicators

  • Percentage Complete: Overall refresh progress
  • Documents Processed: Number of documents handled
  • Processing Rate: Documents per minute/hour
  • Estimated Completion: Projected finish time

Post-Refresh Validation​

Content Verification

  • Document Count: Verify expected number of documents processed
  • Content Quality: Check that content was extracted correctly
  • Search Functionality: Test search results after refresh
  • Performance: Validate search response times

Error Analysis

  • Processing Errors: Review any documents that failed to process
  • Warning Messages: Investigate warnings and potential issues
  • Log Analysis: Review detailed processing logs
  • Resolution Planning: Plan fixes for identified issues

Automated Refresh Strategies​

Scheduled Refresh​

Configuration Options

  • Daily Refresh: For frequently updated content
  • Weekly Optimization: For regular maintenance
  • Monthly Full Refresh: For comprehensive updates
  • Custom Schedules: Based on business requirements

Schedule Management

Example Schedules:
- Product docs: Daily at 2 AM
- Policy documents: Weekly on Sunday
- Web content: Every 6 hours
- Training materials: Monthly on 1st

Resource Planning

  • Off-Peak Scheduling: Run during low-usage periods
  • Resource Allocation: Ensure sufficient system resources
  • Conflict Avoidance: Avoid conflicts with other scheduled operations
  • Priority Management: Prioritize critical knowledge bases

Event-Driven Refresh​

File System Events

  • Real-Time Updates: Process changes as they happen
  • Batch Processing: Group changes for efficient processing
  • Threshold-Based: Trigger refresh when change volume reaches threshold
  • Smart Scheduling: Delay refresh to optimize resource usage

External Triggers

  • API Triggers: Refresh via API calls from external systems
  • Webhook Integration: Respond to external system notifications
  • Manual Triggers: User-initiated refresh operations
  • Conditional Logic: Refresh based on specific conditions

Performance Optimization​

Index Optimization​

Regular Maintenance

  • Index Compaction: Remove fragmentation and optimize structure
  • Statistics Updates: Update search statistics for better performance
  • Cache Optimization: Optimize caching for frequent queries
  • Memory Management: Optimize memory usage for large indexes

Performance Tuning

  • Chunk Size Optimization: Adjust content chunking for better search
  • Embedding Optimization: Optimize vector embeddings for speed
  • Query Optimization: Tune search algorithms and parameters
  • Resource Allocation: Allocate appropriate system resources

Content Optimization​

Quality Improvement

  • Content Curation: Remove low-quality or irrelevant content
  • Duplication Removal: Identify and remove duplicate content
  • Format Standardization: Ensure consistent content formatting
  • Metadata Enhancement: Improve content metadata and tagging

Structure Optimization

  • Document Organization: Optimize document structure and hierarchy
  • Cross-Reference Improvement: Enhance content relationships
  • Categorization: Improve content categorization and classification
  • Access Pattern Optimization: Optimize for common access patterns

Content Management During Refresh​

Version Control​

Content Versioning

  • Change Tracking: Track content changes over time
  • Version History: Maintain history of content versions
  • Rollback Capability: Ability to revert to previous versions
  • Conflict Resolution: Handle conflicting content updates

Backup and Recovery

  • Pre-Refresh Backup: Backup knowledge base before major refreshes
  • Point-in-Time Recovery: Restore to specific points in time
  • Incremental Backups: Regular incremental backups
  • Disaster Recovery: Comprehensive disaster recovery procedures

Content Validation​

Quality Checks

  • Content Completeness: Verify all expected content is present
  • Format Validation: Ensure content is properly formatted
  • Link Validation: Check that internal references are valid
  • Metadata Verification: Validate content metadata and properties

Consistency Checks

  • Cross-Reference Validation: Verify content relationships
  • Terminology Consistency: Check for consistent terminology usage
  • Style Consistency: Ensure consistent content style
  • Information Currency: Verify content is up-to-date

Troubleshooting Refresh Issues​

Common Problems​

Refresh Failures

  • Permission Issues: Check file and directory permissions
  • Resource Constraints: Verify sufficient system resources
  • Network Problems: Check connectivity to external sources
  • Configuration Errors: Validate knowledge base configurations

Performance Issues

  • Slow Processing: Optimize processing parameters and resources
  • Memory Problems: Increase available memory or optimize usage
  • Storage Issues: Ensure sufficient storage space
  • Concurrent Conflicts: Manage concurrent refresh operations

Content Issues

  • Processing Errors: Investigate document processing failures
  • Quality Problems: Address content quality and format issues
  • Missing Content: Verify source accessibility and permissions
  • Duplicate Content: Identify and resolve content duplication

Diagnostic Tools​

Refresh Logs

  • Detailed Processing Logs: Review step-by-step processing information
  • Error Messages: Analyze specific error messages and codes
  • Performance Metrics: Review processing times and resource usage
  • Success Statistics: Validate processing success rates

System Monitoring

  • Resource Usage: Monitor CPU, memory, and storage during refresh
  • Network Activity: Track network usage for remote sources
  • Database Performance: Monitor database operations and performance
  • Index Health: Check search index status and health

Best Practices​

Refresh Planning​

Regular Maintenance Schedule

  • Establish Routine: Create regular refresh schedules
  • Monitor Performance: Track refresh performance over time
  • Plan Resources: Ensure adequate resources for refresh operations
  • Document Procedures: Maintain clear refresh procedures

Change Management

  • Test Refreshes: Test refresh operations in staging environments
  • Gradual Rollout: Implement changes gradually
  • Monitor Impact: Track impact of refresh operations
  • Rollback Planning: Have rollback procedures ready

Performance Management​

Resource Optimization

  • Schedule Optimization: Optimize refresh schedules for resource usage
  • Parallel Processing: Use parallel processing where appropriate
  • Resource Monitoring: Continuously monitor resource usage
  • Capacity Planning: Plan for growing content and usage

Quality Assurance

  • Validation Procedures: Implement comprehensive validation procedures
  • Quality Metrics: Track content quality metrics over time
  • User Feedback: Incorporate user feedback into refresh procedures
  • Continuous Improvement: Continuously improve refresh processes

Security and Compliance​

Access Control

  • Refresh Permissions: Control who can initiate refresh operations
  • Audit Logging: Log all refresh operations and changes
  • Data Security: Ensure data security during refresh operations
  • Compliance Monitoring: Ensure refresh operations meet compliance requirements

Data Protection

  • Backup Procedures: Implement comprehensive backup procedures
  • Recovery Testing: Regularly test recovery procedures
  • Data Integrity: Ensure data integrity during refresh operations
  • Privacy Protection: Protect sensitive data during processing

Regular knowledge refresh ensures your AI workflows have access to current, accurate, and well-organized information. Implement these maintenance practices to keep your knowledge bases performing optimally.