Skip to main content

list_objects_parallel_pages

Function list_objects_parallel_pages 

Source
pub async fn list_objects_parallel_pages(
    client: &Client,
    bucket: &str,
    prefix: &str,
    max_pages: usize,
) -> Result<Vec<ListPage>, AppError>
Expand description

Fetch up to max_pages pages of hierarchical object listing starting at the given prefix.

ยงDesign choice: sequential pre-fetch

AWS continuation tokens are derived from the last key of each page, so the token for page N is only available after fetching page N-1. True parallel pagination would require guessing alphabetic split points, which is fragile and adds little value for the typical page sizes used here. This function therefore fetches pages sequentially in a loop. The speed benefit over calling list_objects in a loop comes from batching the results and returning them in one allocation.

max_pages defaults to 4. Pass 0 to get a single page.