At SpaceKnow, our operations revolve around managing and analyzing Petabytes of satellite imagery. Our platform — primarily driven by Python — delivers world-class geospatial analysis using advanced AI algorithms, including object detection and classification. This colossal task brings along certain challenges unique to large-scale implementations.

This talk will expose and address some of these challenges and provide simple practical examples/hacks that we are employing to keep our sanity.

The solutions to following challenges will be explored:

• Logging: How to smartly utilize Python's logging capabilities to maintain context and traceability of logs in a cloud-based distributed system.
• Versioning: Managing pip and system packages for python running in docker can be tricky, especially when services demand predictable output. I will shed light on the problems faced and the tricks used to effectively manage versioning in our platform.
• Parallelism: Large-scale operations require robust tasking mechanisms. We'll explore how we use Redis to enable parallel Python microservices to run reliable tasking in a distributed system.
• Resource Scaling: Resource scaling is critical for managing load in our applications. We will discuss how we automate the scaling of Kubernetes pods based on queue size using Python microservices.