SNEI(Sony)
Design Principles for Ingestors
Simple
Independent
- Ingesters are independent and can be ran at same time
Idempotent
- Able to rerun ingester (with new change/logic), not impact to clients
- Generate HashId
Testablity
Visibility - monitoring
Design Mistakes
Use Dynamodb to store data
Solr schema for the unique field
Why happened?
Why happened?
- Only considered current requirements, didn't plan/think features that we need in near future, like need query and export data
