Application Insights Backend Monitoring Specification
Purpose & Scope
This specification defines the implementation of Azure Application Insights for backend performance monitoring, error tracking, and system health visibility for the Digital Lock On API services.
Requirements Implemented
- REQ-ANALYTICS-001: Product Analytics and System Monitoring - Backend monitoring requirements
Related Decisions
- [ ] To be created: ADR-XXX: Analytics and Monitoring Tool Selection
Architecture Overview
Application Insights will be integrated into the .NET backend API to provide automatic instrumentation of HTTP requests, dependencies, exceptions, and custom telemetry for business-critical operations.
Integration Points
Automatic Instrumentation:
We will take the approach of implementing the standard configuration, and decrease or increase the scope from there.
We have a scaling policy in place to ensure there are appropriate memory and resources available for the application.
If needed, we can additionally look to implement:
- HTTP requests (all API endpoints)
- Database queries (Entity Framework/Dapper)
- External HTTP calls
- Azure service dependencies
Custom Telemetry (TBC):
- Handover operations
- Sync processes
- QR code operations
- Background jobs
Key Metrics to Track
Automatic Metrics (No Code Required)
From the Cloud, Application Insights automatically captures:
- Request duration and status codes
- Failed request count
- Server response time
- Dependency call duration
- Exception count and types
Dashboards
Live Metrics Dashboard
Real-time monitoring for incident response, with automated alerting at certain thresholds:
- Failed request rate
- Server response time
- Server resource utilisation
- Deployment status
Operational Dashboard
Daily operational health:
- Request volume (last 24 hours)
- P50, P95, P99 response times by endpoint
- Top 5 slowest endpoints
- Exception count by type
- Dependency failure rate
Business Metrics Dashboard
Product and business insights:
- Sync operation success rate (application layer)
- QR code generation success (back end, application layer)
Error Handling
Exception Categorisation
User Errors (Log, Don't Alert):
- Validation failures (400 Bad Request)
- Authentication failures (401 Unauthorized)
- Permission denied (403 Forbidden)
- Not found (404)
System Errors (Log & Alert):
- External service timeouts
- Unhandled exceptions (500)
- Dependency failures
Expected Errors (Log Only):
- Handover timeouts (business logic)
- Offline sync conflicts (resolved automatically)
- QR code email bounces (expected failure mode) - may need to notify administrator
Privacy & Security
PII Handling
Do NOT log:
- Passwords or authentication tokens
- Full email addresses (log hashed or domain only)
- Phone numbers
- Worker personal details
Safe to log:
- User IDs (UUIDs)
- Worksite IDs
- Timestamps
- Operation types
- Success/failure status
- Performance metrics
Maintenance
Regular Reviews
Daily:
- Check alert notifications
- Review critical errors in Live Metrics
Weekly:
- Review slow query report
- Check P95/P99 response time trends
- Validate alert threshold accuracy
Monthly:
- Review and update alert rules
- Archive or delete old custom events if storage grows
- Update performance baselines based on trends
Data Retention
Application Insights default retention:
- 90 days for all telemetry data
- Can be extended to 730 days (additional cost)
Recommendation:
- Keep 90-day default for cost efficiency
- Export critical data to blob storage for long-term retention if needed
- Archive important incidents and post-mortems separately
Cost Management
Free Tier Allowances
- First 5GB data ingestion per month: Free
- Additional data: ~NZD $3.50/GB
Cost Optimisation
At current scale:
- Estimated daily ingestion: ~100-200MB
- Monthly cost: Likely within free tier
If costs increase:
- Enable adaptive sampling (reduce telemetry volume)
- Filter out noisy low-value telemetry
- Reduce retention period
- Sample non-critical endpoints