How to Download All Files from DigitalOcean Spaces Using Python

DigitalOcean Spaces provides a fantastic, S3-compatible object storage solution. However, when it comes to backing up or migrating your data, downloading thousands of files and nested folders manually from the dashboard can be a nightmare.

Fortunately, since DigitalOcean Spaces is fully compatible with the Amazon S3 API, we can use Python's powerful boto3 library to automate the entire download process. In this guide, I'll show you exactly how to write a Python script that recursively downloads all files and folders from your DigitalOcean Space.

Prerequisites

Before we start coding, make sure you have the following:

Python 3.x installed on your machine.
Your Space Name (Bucket name) and the Endpoint URL (e.g., sgp1.digitaloceanspaces.com).
Your Access Key and Secret Key. You can generate these from the API section in your DigitalOcean dashboard.

Step 1: Install Boto3

First, you need to install boto3, which is the AWS SDK for Python. Open your terminal and run:

bash

pip install boto3

Step 2: The Python Script

Create a new file named main.py and paste the following code. This script connects to your Space, recursively iterates through all folders, and downloads every single file into a local ./download directory while maintaining the original folder structure.

python

import boto3
import os

# DigitalOcean Space Configuration
SPACE_NAME = 'your-space-name' # Replace with your bucket name
ACCESS_KEY = 'your-access-key' # Replace with your access key
SECRET_KEY = 'your-secret-key' # Replace with your secret key
ENDPOINT_URL = 'https://sgp1.digitaloceanspaces.com' # Replace with your region endpoint

# Local directory where files will be saved
LOCAL_DIRECTORY = './download'

# Initialize the boto3 client
s3 = boto3.client('s3', endpoint_url=ENDPOINT_URL,
                  aws_access_key_id=ACCESS_KEY,
                  aws_secret_access_key=SECRET_KEY)

def download_files_from_space(space_folder=''):
    """
    Recursively fetches and downloads all files from the specified DigitalOcean Space.
    """
    # List all objects in the current space folder
    response = s3.list_objects_v2(Bucket=SPACE_NAME, Prefix=space_folder)

    # Check if there are any objects found
    if 'Contents' in response:
        objects = response['Contents']

        # Loop through and process each object
        for obj in objects:
            key = obj['Key']
            
            if key.endswith('/'):
                # It's a folder: recursively explore it
                folder_name = key.rstrip('/')
                next_space_folder = os.path.join(space_folder, folder_name)
                download_files_from_space(next_space_folder)
            else:
                # It's a file: prepare the local path and download it
                local_file_path = os.path.join(LOCAL_DIRECTORY, key)
                
                # Ensure the parent directories exist locally
                os.makedirs(os.path.dirname(local_file_path), exist_ok=True)
                
                print(f"Downloading {key}...")
                s3.download_file(SPACE_NAME, key, local_file_path)
                print(f"✅ Successfully downloaded to {local_file_path}")
    else:
        print(f"No objects found in folder: {space_folder}")

if __name__ == "__main__":
    print("Starting download process...")
    download_files_from_space()
    print("🎉 All files downloaded successfully!")

How This Script Works

Initialization: We initialize a boto3.client('s3') but point it to the DigitalOcean endpoint instead of AWS.
Pagination/Listing: list_objects_v2 fetches the contents of the Space. (Note: If you have more than 1000 objects, you will need to handle pagination using ContinuationToken, but this script covers most standard use cases).
Recursive Traversal: The script checks if a Key ends with a /. If it does, it recognizes it as a directory and recursively calls itself.
Directory Mirroring: os.makedirs() ensures that your Space's exact folder structure is perfectly replicated on your local machine before the file is saved.

Running the Script

Once you have replaced the credentials with your own, simply run the script from your terminal:

bash

python main.py

Watch as your terminal lights up, seamlessly pulling down all your cloud assets to your local machine!

Conclusion

Using boto3 to interact with DigitalOcean Spaces (or any S3-compatible API) makes cloud storage management highly automatable and incredibly efficient. This script is perfect for backups, migrations, or local environment setups.

Tech Stack: Python, Boto3, DigitalOcean Spaces, AWS S3 API