ETag Setup for Corpus Incremental Refresh

When a corpus includes web pages, those pages keep changing after you index them: prices update, policies get revised, and new articles appear. Scheduled Refresh re-indexes your web links automatically so assistants never answer from stale content. Incremental Refresh, the most efficient refresh type, depends on a small piece of web technology called an ETag. This page explains what ETags are, why they matter, and how to turn them on for common web servers.

What is an ETag?

An ETag (entity tag) is a short identifier a web server sends with a page, like a fingerprint of its current content. When the content changes, the ETag changes too. By comparing the ETag it saw last time with the one the server returns now, Ejento AI can tell whether a page changed without downloading and re-processing it.

Why it matters for your corpus

Scheduled Refresh offers two refresh types:

Refresh type	What it does	Needs ETags?	Best for
Incremental Refresh	Refreshes only the links that have changed since the last run	Yes	Large sites where most pages rarely change
Full Refresh	Refreshes all links from scratch on every run	No	Small sites, or servers you can't configure

With ETags configured, Incremental Refresh skips unchanged pages. Refreshes finish faster, use less processing, and cost less, while changed pages are still picked up on the next run. Without ETags, Incremental Refresh can't detect changes reliably and won't work as expected. In that case, use Full Refresh instead.

Good to know about Scheduled Refresh

It applies only to web links. Uploaded files, text blobs, and other sources aren't re-indexed by Scheduled Refresh.

You choose how often it runs. Set the Frequency (Daily, Weekly, Monthly, or Yearly) and a Time (Local). The settings show a summary such as "Incremental refresh runs daily at 2:30 AM".

Saving asks you to confirm. When you save with Incremental Refresh selected, Ejento AI reminds you that it relies on ETags and asks you to confirm.

It's set per corpus. Turn it on from the corpus Settings tab. See Assistant Corpus Settings.

Use cases

Your own help center: Your corpus indexes your help center, and you can change its server configuration. Turn on ETags so a daily Incremental Refresh picks up new and changed articles without re-processing the rest.

Large documentation sites: You index many pages that rarely change. ETags let you refresh often while paying only for the pages that changed.

Someone else's website: If the site belongs to someone else and doesn't send ETags, you can't add them yourself. Use Full Refresh for those links.

How to check whether a site sends ETags

Open the page in your browser's developer tools, go to the Network tab, select the page request, and look for an ETag header in the response. You can also run curl -I https://your-site.com/page and look for an ETag: line.

Setting up ETags on common servers

Ask the team that manages your website to apply the configuration that matches your environment.

1. Apache HTTP Server

Enable ETags Based on File Properties

Avoid differences across servers by ignoring inode values.

Source: KeyCDN Guide on ETags

2. Nginx

Enable ETags Globally

Enable ETags for Specific Files (e.g., HTML)

Note: If gzip is enabled, ETags may be stripped.

Source: ServerFault Thread

Advanced: Use Dynamic ETag Module

Automatically generates ETags for dynamic responses.

Source: GetPageSpeed Nginx Extras

3. Node.js / Express (REST APIs)

Enable ETags in Express

Source: Express ETag Documentation

4. Spring Boot (Java)

Add ETag Support with ShallowEtagHeaderFilter

Automatically adds ETags based on response content.

Source: Implementing ETags in REST APIs – Medium

5. Cloudflare

Cloudflare Behavior

Cloudflare preserves ETags from the origin but does not generate them.

If content is cached at Cloudflare, ETags may not be revalidated.

Best Practices:

Ensure your origin server sends ETag headers.

Use:

to force revalidation and allow conditional requests using ETags.

Summary

Environment	Method	Notes
Apache	`FileETag` + `Header`	Use MTime and Size for consistency
Nginx	`etag on` or dynamic module	Gzip can interfere with ETags
Express (Node.js)	`app.set('etag', 'strong')`	Built-in support in Express
Spring Boot	`ShallowEtagHeaderFilter`	Adds ETags for REST responses
Cloudflare	Preserve origin ETags	Requires correct `Cache-Control`

ETag Setup for Corpus Incremental Refresh

What is an ETag?#

Why it matters for your corpus#

Good to know about Scheduled Refresh#

Use cases#

How to check whether a site sends ETags#

Setting up ETags on common servers#

1. Apache HTTP Server#

Enable ETags Based on File Properties#

2. Nginx#

Enable ETags Globally#

Enable ETags for Specific Files (e.g., HTML)#

Advanced: Use Dynamic ETag Module#

3. Node.js / Express (REST APIs)#

Enable ETags in Express#

4. Spring Boot (Java)#

Add ETag Support with ShallowEtagHeaderFilter#

5. Cloudflare#

Cloudflare Behavior#

Summary#