SFTP server guidelines

📘

Note:

By using this API and its documentation and building an integration, you agree to the Additional API Terms and Guidelines.

Introduction

Indeed’s SFTP server is a shared resource that enables clients to upload their XML files to Indeed. After the client uploads XML files, Indeed processes the files so that the jobs appear on our website.

🚧

Warning:

Because the SFTP server is a shared resource, it is important for all clients to follow these guidelines. Following these guidelines enables Indeed to process all XML files in a timely manner and ensures that all jobs are posted to Indeed. If a client violates these guidelines, they risk having their access to the server revoked.

1) Use the minimal number of connections needed

Indeed limits the number of connections to mitigate DDOS attacks. Limit the number of connections to the amount needed so that every client can successfully upload their files.

Ideally, each client should use only one connection. Indeed knows that some clients upload multiple files concurrently, which might mean multiple connections. By using the SFTP protocol, one connection can multiplex operations. This action enables the client to complete multiple, disparate operations in a single connection.

2) Just upload the file with the original filename

Indeed starts to process the file with the original filename as soon as it is uploaded. Changing the filename can delay processing time by up to 10 minutes.

Do not:

• Change the filename after uploading the file.
• Delete the file after uploading it.
• Manipulate the file metadata, such as the modification time or permissions.
• Constantly query/poll the state of the files.

Indeed provides a two-day look-back in your directory under the processed subdirectory. This look-back tells you whether the file uploaded successfully.

For example: If the file was uploaded to ftp/ftpexample123/file.xml on July 19, 2022, at 10:31:00 Central Daylight Time, Indeed uploads an empty file at ftp/ftpexample123/processed/file.xml/20220719_103100_CDT.xml. Use this file as the acknowledgment that the file was uploaded and is being processed. This file persists for two days before being deleted.

3) Determine upload frequency

There are two different categories you can use to determine how frequently to upload: time-based or significant-changes-based in the XML.

• Time-based:
  • Guideline: Every 6 hours
  • Minimum amount of time between uploads: 2 hours
• Significant-changes-based:
  • Guideline: Update when changing at least 1% of jobs.

  • 📘 Note: Changing the URL for many jobs does not constitute a significant change.

Our goal is to serve jobseekers with the most accurate and timely updates of jobs as possible. However, there are a number of steps that jobs go through before they appear on Indeed.com. Uploading every 15 minutes with minor changes does not make jobs appear faster. before the job surfaces, Indeed must reprocess the entire XML file and the job must continue through the rest of Indeed’s job pipeline. Our systems are configured to limit accounts that frequently upload and to only pick up updates every three hours, which is subject to change.

4) Upload necessary files only

After each file uploads, Indeed checks the file name to determine whether Indeed can handle that file name. If Indeed does not know how to handle the file name, Indeed considers the file name unnecessary and moves it to an inaccessible directory.

Uploading unnecessary files adds additional load on the system and takes away from processing files that matter.

If requested, Indeed can query your uploaded files to see which files are not being processed.

5) Follow naming conventions

5.1) Keep the same filename between uploads

Use a consistent name between uploads to make it easier to keep track of files (e.g. my_jobs.xml)

The SFTP server can handle changing filenames between uploads as long as the only change is for datetime.

For example, if I upload the my_jobs_20220816_064321.xml file and, six hours later, upload the my_jobs_20220816_124321.xml file, they are stored as:

• my_jobs_20220816_064321.xml/20220816_064353.xml
• my_jobs_20220816_124321.xml/20220816_124403.xml

5.2) Include a file extension

Indeed only processes XML files at this time, so all files are assumed to be XML files. In the future, Indeed might process other file types. To reduce future issues, include a file extension.

• Incorrect file names examples:
  • MY_JOBS
  • INDEED_JOBS_XML
• Correct file name examples:
  • my_jobs.XML
  • INDEED_jobs.xml