Why we made Shapefile conversion async

Converting a Shapefile, File Geodatabase, GeoPackage, or KML to GeoJSON can take anywhere from a few hundred milliseconds to several minutes, depending on the file. Holding an HTTP request open for that is a bad idea, so /gis/convert-to-geojson/ doesn't convert anything itself: it checks the upload, hands the work to a queue, and answers right away with a job.

The shape of it

Send the files and their format:

curl -X POST "https://geosln-api.com/gis/convert-to-geojson/" \
  -H "Authorization: Token $GEOSLN_API_TOKEN" \
  -F "format_type=shapefile" \
  -F "[email protected]"

You get 202 Accepted and a job, with 1 credit reserved for it:

{
  "id": "7f3c…",
  "kind": "convert",
  "status": "queued",
  "progress": { "done": 0, "total": 0 },
  "credits_reserved": 1,
  "credits_charged": 0,
  "status_url": "https://geosln-api.com/gis/jobs/7f3c…/"
}

Poll status_url until status is succeeded (or failed), then download the result:

curl -OJ "https://geosln-api.com/gis/jobs/7f3c…/result/" \
  -H "Authorization: Token $GEOSLN_API_TOKEN"

The credit is only charged if the conversion succeeds; a failed or cancelled job gives it back. Results are kept for 24 hours.

Behind that, the job sits in a Redis-backed queue (RQ), and dedicated conversion workers run GDAL on it, separate from the web server. A large upload keeps a worker busy, not the API.

Validating before it hits the queue

Everything that can be checked cheaply is checked before the job is queued: the file names and sizes (up to 95MB per upload, and a cap on what a zip unpacks to), that the zip holds what the format needs, and that the layer you asked for exists. A malformed upload fails fast with a 400 and a message that says what's wrong, and nothing is charged. It doesn't take up a worker only to fail a minute later.

# Simplified from the real check, which runs per format
def validate_gis_file_layer(input_file, format_type, layer_name=None):
    available_layers = get_layer_names(input_file, format_type)
    if layer_name and layer_name not in available_layers:
        raise CustomAPIException(
            f"Layer '{layer_name}' not found. "
            f"Available layers: {', '.join(available_layers)}"
        )

This is the same pattern we'd recommend for any pipeline that ingests user-supplied geospatial data: validate cheaply and synchronously, then queue the expensive work, and give the client one place to check on it. Our bulk flood risk endpoint works the same way.