Sitecore XP fundamentals and day-to-day solutions—admin tasks, forms, Helix practices, and real project troubleshooting.

Sitecore

Sitecore XM Cloud guides—publishing, serialization, GraphQL, Experience Edge, workflows, and cloud-native Sitecore delivery.

Sitecore XM Cloud

Sitecore Search tutorials—API crawlers, feed crawlers, SFTP indexing, pagination strategies, and search configuration.

Sitecore Search

Sitecore Content Serialization (SCS) patterns for XM Cloud and XP— module.json setup, sync workflows, and multi-site serialization.

Content Serialization

Sitecore Forms with webhooks, Sitecore Connect, and practical form integration patterns for modern Sitecore solutions.

Sitecore Forms

Sitecore Content Hub and DAM connector guides—browse, search, and insert digital assets into XM Cloud and Experience Editor.

Content Hub / DAM

Community highlights, Stack Exchange contributions, blogging milestones, and Sitecore ecosystem engagement.

Community

When working with APIs that return a lot of data, it’s not practical (or efficient) to get everything in one giant response. That’s where pagination comes in. Sitecore supports two main ways of handling this: offset-based and cursor-based pagination. Let’s break these down and see how you can use them in Sitecore.

What Is Pagination?

Imagine you’re browsing a huge list of users or products. Instead of loading all 10,000 items at once, the system gives you smaller "pages"—maybe 100 at a time. This is pagination, and it makes things faster and easier to manage.

Offset-Based Pagination

How it works:

Offset-based pagination is like flipping through pages in a book. You ask for page 1, then page 2, then page 3, and so on.

Example API call:

https://example.com/api/users?pageSize=100&currentPage=1

Sample API Response:

{
  "totalItems": 1000,
  "pageSize": 100,
  "currentPage": 1,
  "data": [ ... ]
}

How to set it up in Sitecore:

In Sitecore, you can write a Request Extractor to automatically build the URLs for the next pages:

function extract(request, response) {
    const totalItems = response.body?.totalItems;
    const pageSize = response.body?.pageSize;
    const currentPage = response.body?.currentPage;

    const totalPages = Math.ceil(totalItems / pageSize);
    const urls = [];

    for (let i = currentPage + 1; i <= totalPages; i++) {
        urls.push({
            method: 'GET',
            url: https://example.com/api/users?pageSize=${pageSize}&currentPage=${i},
            headers: request.headers
        });
    }

    return urls;
}

This script figures out how many more pages there are and builds the requests for each one.

Cursor-Based Pagination

How it works:

Instead of counting pages, the server tells you where to go next using a "cursor"—usually a token like nextPageToken or lastItemId.

Example API call:

https://example.com/api/users?cursor=abc123

Sample API Response:

{
  "nextPageToken": "abc123",
  "data": [ ... ]
}

How to set it up in Sitecore:

You can use this simple Request Extractor:

function extract(request, response) {
    const cursor = response.body?.nextPageToken;

    if (cursor) {
        return [{
            method: 'GET',
            url: https://example.com/api/users?cursor=${cursor},
            headers: request.headers
        }];
    }
    return [];
}

If the API provides a cursor, Sitecore queues the next request using that token. When there's no more cursor, you're done.

Extra Tip: Using Request Extractors and URL Matchers

Sometimes, you might need to go beyond just pagination—for example, fetching details for each user in the list. That’s where Request Extractors and URL Matchers come in handy. They help you build additional requests based on the API response and filter out the ones you don’t need.

  

Happy Coding!


Pagination in Sitecore: Understanding Offset vs. Cursor

Implementing a Feed Crawler in Sitecore with SFTP body { font-family: Arial, sans-serif; line-height: 1.6; margin: 20px; padding: 20px; } h1, h2, h3 { color: #333; } code { background-color: #f4f4f4; padding: 3px; border-radius: 4px; } pre { background: #f4f4f4; padding: 10px; border-radius: 5px; overflow-x: auto; }

Setting Up a Feed Crawler in Sitecore with SFTP

Introduction

A feed crawler is essential for fetching and indexing external data sources into Sitecore. Whether integrating product catalogs, news articles, or structured data from an external system, Sitecore’s feed crawler enables seamless content synchronization. This guide will walk you through setting up a feed crawler in Sitecore using SFTP as the data source.

Prerequisites

Access to Sitecore CMS (XM Cloud or XP)
Knowledge of Sitecore Content Search and Indexing
SFTP server credentials (hostname, username, password, or SSH key)
External data source (CSV, JSON on an SFTP server)

Step 1: Define the Feed Source

Before setting up the crawler, ensure that the feed file is structured correctly and accessible via SFTP.

Example JSON Feed Structure

{"id": "1", "name": "Harry Potter", "city": "London", "type": "Test"}
{"id": "2", "name": "Don Quixote", "city": "Madrid", "type": "Test"}
{"id": "3", "name": "Joan of Arc", "city": "Paris", "type": "Test"}
{"id": "4", "name": "Rosa Parks", "city": "Alabama", "type": "Test"}
{"id": "5", "name": "Ram", "city": "London", "type": "Test"}
{"id": "6", "name": "Don", "city": "Madrid", "type": "Test"}
{"id": "8", "name": "Joan D", "city": "Madrid", "type": "Test"}
{"id": "10", "name": "Rosa Don", "city": "Emeia", "type": "Test"}

Example SFTP File Structure

/upload/
    test.json

Step 2: SFTP Upload Directory Setup

We have successfully created the "upload" directory, which will act as the upload location for the feed files.

Step 3: Configure the Sitecore Feed Crawler with SFTP

Navigate to Sitecore Search Admin > Crawlers.
Click Add New Crawler.
Select Feed Crawler.
Configure the feed source:
    Protocol: SFTP
    Host: sftp.example.com
    Port: 22
    Username & Password (or SSH Key Authentication)
    Remote Path: /upload/test.json
    Schedule: Daily, Hourly, or Custom Interval
Define Data Mapping to match feed fields to Sitecore fields.
Save and start the crawl.

Step 4: Ensuring Proper Feed Formatting

Feed files should be structured as one JSON object per line.
The document extractor function should return an array of objects.



Example JavaScript Extractor:

return [JSON.parse(line)];

Step 5: Troubleshooting Indexing Errors

If you encounter an error such as:

Internal Server Error
Validation Error: The number of indexed documents should be bigger than 0

Follow these steps:

Ensure that the JSON file is correctly formatted.
Verify that the file is uploaded to the correct directory.
Check the Document Extractor settings.
Confirm that transformers are properly configured.

Limitations

The Analytic Dashboard currently does not support Feed Crawler visibility. A feature request has been submitted for future consideration.

Conclusion

Using SFTP for feed crawling in Sitecore allows for secure and automated content ingestion. By leveraging Sitecore Search organizations can efficiently manage external data sources while maintaining data security.


Implementing a Feed Crawler in Sitecore Search with SFTP

Mastering Sitecore Search API Crawlers

Sitecore Search is an integral part of the Sitecore Experience Platform (XP) that helps businesses deliver highly personalized and relevant search experiences. One of the most powerful features of Sitecore Search is its ability to index external data sources, such as APIs, to provide dynamic content in search results. In this guide, we’ll explore the ins and outs of using Sitecore's API Crawler—a tool that helps index API data—and show you how to implement it for seamless integration.

Whether you're a developer new to Sitecore or someone looking to improve your Sitecore Search implementation, this guide will take you through essential configurations, indexing scenarios, and advanced pagination techniques that will help you build powerful search experiences.

Table of Content

What is Sitecore Search and API Crawling?

Key Concepts You Should Know

Setting Up Sitecore API Crawler for Basic Indexing

Understanding Pagination: Offset vs. Cursor

How to Implement Offset-Based Pagination

How to Implement Cursor-Based Pagination

Using Request Extractors and URL Matchers

Advanced Techniques: Handling MAX DEPTH and Data Indexing

Conclusion

What is Sitecore Search and API Crawling?

Sitecore Search is a powerful search engine built into the Sitecore Experience Platform. It helps manage content and deliver personalized, relevant search results to your users. With Sitecore Search, you can index content from various data sources, including Sitecore-managed content and external APIs.

When you need to index content from an API, API Crawlers come into play. API Crawlers allow Sitecore Search to communicate with external APIs, extract data, and make it searchable on your site. This is particularly useful when your content resides in third-party systems or databases, rather than being directly managed within Sitecore.

Key Concepts You Should Know

Before we dive deeper into the configurations, let’s review some essential concepts that will help you understand how Sitecore’s API Crawling works:

Source: A source in Sitecore defines the connection between the crawler and your data. For instance, you might have an API source configured to pull data from a third-party API.
    
Attributes: These are the data fields within your API response that you want to index. For example: id, name, street, city, state, zipcode, phone\_number, extension , webaddress, sites, type etc.
    
Indexing: This refers to the process of retrieving data from your source (API) and making it available for search. Sitecore will crawl the data, extract the relevant information, and build an index.
    
Request Extractor: A tool that dynamically constructs URLs based on your API response. This is especially useful when you're dealing with paginated data.
    
Pagination: When your API data is spread across multiple pages, you need a way to retrieve each page in sequence. There are two main types of pagination you can use in Sitecore: offset-based and cursor-based pagination.
Setting Up Sitecore API Crawler for Basic Indexing

Let's start by setting up a simple API crawler that doesn’t require pagination. This means we’ll retrieve all the data from an API in a single call.

Example API Response (Without Pagination):

Imagine an API that returns user data in the following structure:

{
  "totalItems": 20,
  "pageSize": 100,
  "currentPage": 1,
  "data": [    {        "Orgid": 1,        "name": "Tech Solutions HQ",        "street": "123 Innovation Drive",        "city": "San Francisco",        "state": "CA",        "zip": "94107",        "phone": "(415) 555-1234",        "extension": "101",        "website": "https://www.techsolutions.com",        "sites": [            "Main Office",            "West Coast Branch"        ],        "type": "Corporate Office"    },    {        "Orgid": 2,        "name": "Digital Labs",        "street": "456 AI Lane",        "city": "New York",        "state": "NY",        "zip": "10001",        "phone": "(212) 555-5678",        "extension": "202",        "website": "https://www.digitallabs.com",        "sites": [            "Research Center",            "Innovation Hub"        ],        "type": "R&D Facility"    },    {        "Orgid": 3,        "name": "Green Energy Solutions",        "street": "789 Solar Road",        "city": "Austin",        "state": "TX",        "zip": "73301",        "phone": "(512) 555-7890",        "extension": "303",        "website": "https://www.greenenergy.com",        "sites": [            "Solar Farm",            "Wind Turbine Field"        ],        "type": "Energy Company"    }
  ]
}



This API returns a list of users, with their ID, name, street, city, state, zipcode, phone_number, extension , webaddress, sites, type. Since this example does not use pagination, we can index all the users at once.Step 1: Create Custom Attributes in Sitecore SearchBefore indexing the data, you need to define custom attributes in Sitecore Search. These attributes represent the data you want to index. For this example, you’ll need attributes like name, street, city, state, zipcode, phone_number, extension , webaddress, sites, type.name(String)street(String)city(String)state(String)zipcode(String)phone_number(String)extension(String)webaddress(String)sites(Strings)type(String)To add these attributes to the system, you must be assigned the TechAdmin role. Navigate to the Administration menu in the sidebar, go to the Attributes tab, and click the Add Attribute button.In the dialog, configure the following fields: Entity, Display Name, Attribute Name (lowercase without spaces), Data Type, and optionally check the Return in API response checkbox to make it visible in search results.In the Use For Features view, you can enable additional features for the attribute, such as faceting and sorting options. When adding the AreaOfInterest attribute, be sure to check the Facets option. Repeat these steps for each field, and don’t forget to publish your changes by clicking the Publish button in the attribute list view.Source Creation in Sitecore SearchTo set up a source in Sitecore Search, you need to review three key documents from the official documentation:SourcesCreate a SourceConfiguring an API CrawlerAdding a SourceNavigate to the Sources view in the sidebar.Click the Add Source button.Configuring the SourceEnter a Source Name and Description.Select API Crawler as the connector.You can customize your source with various settings, such as Authentication and Scan Frequency, to match your needs.Configuring a Trigger in Sitecore SearchA Trigger defines the starting point for the crawler to begin indexing content.Setting Up a TriggerGo to the Trigger section and click Edit.
There are different types of triggers available, allowing you to configure details like the Request Body or Headers.For this example, we’ll keep it simple and use a GET request without additional configuration.Click Save to apply the changes.Configuring the Document ExtractorSince we’re not using a paging mechanism in this setup, we only need to configure a Document Extractor—no need for a Request Extractor.Locate the Document Extractors section and click Edit.Set a Name for the extractor and select JS as the Extractor Type.We don’t need URL Matchers for this case.Why use a JS Extractor instead of JSONPath?JSONPath extractors work well for indexing a single document per request but come with some limitations.A JS Extractor allows for more flexibility, such as transforming string-based dates into timestamps.Once configured, save your changes to complete the setup.Step 2: Configure the API Crawler SourceNext, configure the API Crawler source in Sitecore to start crawling the data. You’ll need to:Define the Source Name: A descriptive name for your API source, like "UserAPI".Set the Trigger: This is the starting point for your crawl. It should be a GET request to the URL of your API, like https://example.com/api/users.Once this is done, Sitecore will start indexing all the data from this API.Step 3: Extract the DataNow, you need to define how Sitecore should extract relevant data from the API response. This is where the Document Extractor comes in. You can use a JavaScript extractor to map the data to the attributes you defined earlier.Here’s an example of a simple JS Extractor:function extract(request, response) {  let requests = [];  const data = response.body;  if (    data  ) {    requests = data?.map((e, i) => {      return {        "id": e.Orgid,        "name": e.name,        "street": e.street,        "city": e.city,        "state": e.state,        "zipcode": e.zip,        "phone_number": e.phone,        "extension": e.extension,        "webaddress": e.website,        "sites": e.sites,        "type": "Level Of Care"      };    });  }  else {    return {      "description": response.body.data,      "id": "ename",      "url": "response.body.data.item.path",      "type": "ERROR",    };  }  return requests;}Saving and Publishing Your SourceAfter configuring the trigger and document extractor, follow these final steps:Click Save to apply your changes.Publish your source to initiate the indexing process.Wait a few minutes for the indexing to complete.If everything is set up correctly, you should see the indexed data in the Source view.To view your indexed documents, navigate to Content Collection in the sidebar. There, you should see all the documents that have been successfully indexed.

I hope this blog post makes it easier for you to set up and use an API Crawler in Sitecore Search!


Sitecore Search

Pagination in Sitecore: Understanding Offset vs. Cursor

Implementing a Feed Crawler in Sitecore Search with SFTP

Mastering Sitecore Search API Crawlers: A Comprehensive Series (Part 1-3)