Failure Drills

Scenario

Simulate a node going down (hardware failure, OOM kill, network partition) and observe cluster behavior. This is the most common failure mode in production — a support engineer needs to know exactly what happens and how to guide a customer through recovery.

Command

# Kill es02 abruptly (simulates hardware failure)
docker compose stop es02

# Check cluster health immediately
curl -s https://localhost:9200/_cluster/health \
  --cacert ca.crt -u elastic:$PASSWORD | jq

# Check unassigned shards
curl -s https://localhost:9200/_cat/shards?v\&h=index,shard,prirep,state,node \
  --cacert ca.crt -u elastic:$PASSWORD | grep UNASSIGNED

# Recover: restart the node
docker compose start es02

# Watch recovery
curl -s https://localhost:9200/_cat/recovery?v\&active_only \
  --cacert ca.crt -u elastic:$PASSWORD

Observation

Cluster immediately transitioned from GREEN to YELLOW. 22 replica shards that were assigned to es02 became UNASSIGNED. Primary shards remained available on es01 and es03, so all search and indexing operations continued without interruption. The cluster logged master election activity but since es01 and es03 maintained quorum (2 out of 3), no master re-election was needed.

Resolution

After restarting es02, the node rejoined the cluster within 15 seconds. Elasticsearch detected that es02 had a recent copy of the shard data (synced allocation IDs matched) and performed peer recovery — only replaying the transaction log rather than copying entire segments. Cluster returned to GREEN in under 30 seconds.

Scenario

Index a document with a field type that conflicts with the existing mapping. This is one of the most common support tickets — customers have multiple data sources sending different types for the same field name.

Command

# First, index a document with age as a long (number)
curl -X POST "https://localhost:9200/test-mapping/_doc/1" \
  --cacert ca.crt -u elastic:$PASSWORD \
  -H 'Content-Type: application/json' -d '{
    "name": "Kim Minji",
    "age": 28
  }'

# Now try to index age as a string — this will fail
curl -X POST "https://localhost:9200/test-mapping/_doc/2" \
  --cacert ca.crt -u elastic:$PASSWORD \
  -H 'Content-Type: application/json' -d '{
    "name": "Park Sooyoung",
    "age": "twenty-five"
  }'

Observation

The second indexing request returned HTTP 400 with a document_parsing_exception: "failed to parse field [age] of type [long] in document with id 2. Preview of field value: twenty-five". Elasticsearch auto-detected the age field as type long from the first document, and the mapping is immutable — you cannot change a field type once set.

Resolution

Three options for the customer: (1) Reindex into a new index with the correct mapping defined upfront. (2) Use a multi-field mapping with both long and keyword sub-fields. (3) Use an ingest pipeline with a convert processor to normalize the data before indexing. The root cause is usually missing explicit mappings — I'd recommend always defining mappings upfront in an index template rather than relying on dynamic mapping.

Scenario

Delete an entire index (simulating accidental deletion or data corruption) and restore it from a previously created snapshot. This validates the backup/restore pipeline end-to-end.

Command

# Verify snapshot exists
curl -s https://localhost:9200/_snapshot/maclab-backup/snapshot_1 \
  --cacert ca.crt -u elastic:$PASSWORD | jq '.snapshots[0].state'
# → "SUCCESS"

# Delete the support-tickets index
curl -X DELETE "https://localhost:9200/support-tickets" \
  --cacert ca.crt -u elastic:$PASSWORD
# → {"acknowledged": true}

# Confirm deletion
curl -s "https://localhost:9200/support-tickets/_count" \
  --cacert ca.crt -u elastic:$PASSWORD
# → 404 index_not_found_exception

# Restore from snapshot
curl -X POST "https://localhost:9200/_snapshot/maclab-backup/snapshot_1/_restore" \
  --cacert ca.crt -u elastic:$PASSWORD \
  -H 'Content-Type: application/json' -d '{
    "indices": "support-tickets",
    "rename_pattern": "",
    "rename_replacement": ""
  }'

# Verify restoration
curl -s "https://localhost:9200/support-tickets/_count" \
  --cacert ca.crt -u elastic:$PASSWORD | jq '.count'
# → 10

Observation

The index was completely removed — all primary and replica shards deleted. The snapshot restore created a new index with identical settings (including the Korean analyzer configuration) and restored all 10 documents. The restore operation took under 2 seconds for this small index.

Resolution

All 10 bilingual support tickets were restored with their original mappings, settings, and data intact. The Korean analyzer configuration (Nori tokenizer with decompound_mode: mixed) was preserved because snapshot/restore captures the full index metadata. A post-restore search for '클러스터 상태' returned the same 3 results as before deletion.