At SpaceKnow, our operations revolve around managing and analyzing Petabytes of satellite imagery. Our platform — primarily driven by Python — delivers world-class geospatial analysis using advanced AI algorithms, including object detection and classification. This colossal task brings along certain challenges unique to large-scale implementations.
This talk will expose and address some of these challenges and provide simple practical examples/hacks that we are employing to keep our sanity.
The solutions to following challenges will be explored:
- Logging: How to smartly utilize Python's logging capabilities to maintain context and traceability of logs in a cloud-based distributed system.
- Versioning: Managing pip and system packages for python running in docker can be tricky, especially when services demand predictable output. I will shed light on the problems faced and the tricks used to effectively manage versioning in our platform.
- Parallelism: Large-scale operations require robust tasking mechanisms. We'll explore how we use Redis to enable parallel Python microservices to run reliable tasking in a distributed system.
- Resource Scaling: Resource scaling is critical for managing load in our applications. We will discuss how we automate the scaling of Kubernetes pods based on queue size using Python microservices.
What do you need to know to enjoy this talk
Medium knowledge: You use frameworks and third-party libraries.
About the topic
No previous knowledge of the topic is required, basic concepts will be explained.