ETag Setup for Corpus Incremental Refresh

The Incremental Refresh feature in Corpus relies on correctly configured ETags.
ETags (Entity Tags) help identify changes in resources. If the ETag value of a URL remains unchanged, Corpus skips re-indexing that content.
Without proper ETag setup, Incremental Refresh will not function as intended.

This guide provides ETag setup instructions for various environments.

1. Apache HTTP Server

Enable ETags Based on File Properties

Avoid differences across servers by ignoring inode values.

Source: KeyCDN Guide on ETags

2. Nginx

Enable ETags Globally

Enable ETags for Specific Files (e.g., HTML)

Note: If gzip is enabled, ETags may be stripped.

Source: ServerFault Thread

Advanced: Use Dynamic ETag Module

Automatically generates ETags for dynamic responses.

Source: GetPageSpeed Nginx Extras

3. Node.js / Express (REST APIs)

Enable ETags in Express

Source: Express ETag Documentation

4. Spring Boot (Java)

Add ETag Support with ShallowEtagHeaderFilter

Automatically adds ETags based on response content.

Source: Implementing ETags in REST APIs – Medium

5. Cloudflare

Cloudflare Behavior

Cloudflare preserves ETags from the origin but does not generate them.

If content is cached at Cloudflare, ETags may not be revalidated.

Best Practices:

Ensure your origin server sends ETag headers.

Use:

to force revalidation and allow conditional requests using ETags.

Summary Table

Environment	Method	Notes
Apache	`FileETag` + `Header`	Use MTime and Size for consistency
Nginx	`etag on` or dynamic module	Gzip can interfere with ETags
Express (Node.js)	`app.set('etag', 'strong')`	Built-in support in Express
Spring Boot	`ShallowEtagHeaderFilter`	Adds ETags for REST responses
Cloudflare	Preserve origin ETags	Requires correct `Cache-Control`

ETag Setup for Corpus Incremental Refresh

1. Apache HTTP Server#

Enable ETags Based on File Properties#

2. Nginx#

Enable ETags Globally#

Enable ETags for Specific Files (e.g., HTML)#

Advanced: Use Dynamic ETag Module#

3. Node.js / Express (REST APIs)#

Enable ETags in Express#

4. Spring Boot (Java)#

Add ETag Support with ShallowEtagHeaderFilter#

5. Cloudflare#

Cloudflare Behavior#

Summary Table#