SFTP server guidelines

Guidelines for using Aggregation SFTP Server.

 

legal notice

• By using this API and its documentation and building an integration, you agree to the Additional API Terms and Guidelines.

Introduction

Indeed’s SFTP server is a shared resource that lets clients upload XML files to Indeed. After a client uploads XML files, Indeed processes them so that the jobs appear on our website.

Because the SFTP server is shared, all clients must follow these guidelines. These guidelines help Indeed process XML files on time and ensure that jobs are posted to Indeed. If a client does not follow these guidelines, Indeed might revoke that client's access to the server.

Step 1. Use the minimal number of connections needed

Indeed limits the number of connections to reduce DDOS attacks. Use only the number of connections needed so that every client can upload files successfully.

Ideally, each client uses only one connection. Indeed recognizes that some clients upload multiple files at the same time, which can require multiple connections. The SFTP protocol lets one connection multiplex operations. This lets the client complete multiple separate operations through one connection.

Step 2. Just upload the file with the original filename

Indeed starts processing the file with the original filename as soon as the upload finishes. Changing the filename can delay processing by up to 10 minutes.

Do not:

• Change the filename after uploading the file.
• Delete the file after uploading it.
• Change file metadata, such as the modification time or permissions.
• Continuously query or poll the state of the files.

Indeed provides a two-day lookback in the processed subdirectory of your directory. This lookback shows whether the file uploaded successfully.

For example: If the file is uploaded to ftp/ftpexample123/file.xml on July 19, 2022, at 10:31:00 Central Daylight Time, Indeed uploads an empty file at ftp/ftpexample123/processed/file.xml/20220719_103100_CDT.xml. Use this file as confirmation that the file was uploaded and is being processed. This file remains for two days before Indeed deletes it.

File directory where Indeed uploads an empty file

Step 3. Determine upload frequency

Use one of two categories to decide how often to upload: time-based or significant-changes-based in the XML.

• Time-based:

  • Guideline: Every 6 hours
  • Minimum time between uploads: 2 hours

• Significant-changes-based:

  • Guideline: Update when at least 1% of jobs change.

  • Changing the URL for many jobs does not count as a significant change.

Our goal is to give job seekers the most accurate and timely job updates possible. However, jobs go through several steps before they appear on Indeed.com. Uploading every 15 minutes with minor changes does not make jobs appear faster. Before a job appears, Indeed must reprocess the entire XML file, and the job must continue through the rest of Indeed’s pipeline. Our systems are configured to limit accounts that upload too often and to pick up updates every three hours. This timing can change.

Step 4. Upload necessary files only

After each file upload, Indeed checks the filename to determine whether it can process that file. If Indeed does not recognize the filename, Indeed treats the file as unnecessary and moves it to an inaccessible directory.

Uploading unnecessary files adds load to the system and reduces processing capacity for files that matter.

If requested, Indeed can review your uploaded files to identify files that are not being processed.

Step 5. Follow naming conventions

Step 5.1. Keep the same filename between uploads

Use a consistent filename between uploads to make files easier to track, such as my_jobs.xml.

The SFTP server can handle changing filenames between uploads if the only change is the datetime.

For example, if you upload my_jobs_20220816_064321.xml and, six hours later, upload my_jobs_20220816_124321.xml, the files are stored as:

• my_jobs_20220816_064321.xml/20220816_064353.xml
• my_jobs_20220816_124321.xml/20220816_124403.xml

Step 5.2. Include a file extension

Indeed currently processes only XML files, so all files are assumed to be XML. In the future, Indeed might process other file types. To reduce future issues, include a file extension.

• Incorrect filename examples:

  • MY_JOBS

  • INDEED_JOBS_XML

• Correct filename examples:

  • my_jobs.XML

  • INDEED_jobs.xml

 

Was this page helpful?