Prometheus Fails to Start

For days now I've been trying to start Prometheus with Helm chart v33. It starts loading WAL segments, and then, either it succeeds but then stops and restarts, or, like below, it receives the stop signal while loading WALs. And then at some points finishes, but still it restarts.

Most of the times it finishes without interruption and then restarts. This time, it also received a SIGTEM while processing WALs.

I have given it 25GB of memory request, and at the time of the mid-way SIGTERM it was taking 0.14 CPU core and 18GB of memory. So it was not starved.

Now that I describe the pod I see this:

Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Normal   Pulled     13m (x4 over 70m)   kubelet  Container image "" already present on machine
  Warning  Unhealthy  7s (x232 over 69m)  kubelet  Startup probe failed: HTTP probe failed with statuscode: 503

The stateful set has a startup probe set to 60 failures and checks every 15sec, so 15min total. I suspect it fails because the WAL processing takes too long (18+min), but the values file has no place for me to increase the length of the startup probe.

I hope someone can shed some light.

ts=2022-04-01T17:59:19.495Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1648785600000 maxt=1648792800000 ulid=01FZHZ8CF2ZY6D8NRG7YPSNFBC
ts=2022-04-01T17:59:19.496Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock
ts=2022-04-01T17:59:19.497Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-04-01T17:59:19.638Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-04-01T17:59:21.550Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.911956954s
ts=2022-04-01T17:59:21.550Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
 ts=2022-04-01T18:00:11.960Z caller=head_wal.go:337 level=warn component=tsdb msg="Unknown series references" samples=7 exemplars=0
ts=2022-04-01T18:00:11.960Z caller=head.go:569 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2022-04-01T18:00:16.208Z caller=head_wal.go:337 level=warn component=tsdb msg="Unknown series references" samples=218 exemplars=0
ts=2022-04-01T18:00:16.209Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=392 maxSegment=438
ts=2022-04-01T18:00:17.033Z caller=head_wal.go:337 level=warn component=tsdb msg="Unknown series references" samples=91 exemplars=0
ts=2022-04-01T18:00:17.033Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=393 maxSegment=438
ts=2022-04-01T18:03:46.908Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=394 maxSegment=438
ts=2022-04-01T18:04:23.464Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=395 maxSegment=438
ts=2022-04-01T18:04:57.876Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=396 maxSegment=438
ts=2022-04-01T18:06:03.022Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=397 maxSegment=438
ts=2022-04-01T18:06:04.627Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=398 maxSegment=438
ts=2022-04-01T18:06:05.935Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=399 maxSegment=438
ts=2022-04-01T18:07:32.479Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=400 maxSegment=438
ts=2022-04-01T18:08:11.767Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=401 maxSegment=438
ts=2022-04-01T18:08:42.376Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=402 maxSegment=438
ts=2022-04-01T18:10:00.038Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=403 maxSegment=438
ts=2022-04-01T18:10:22.480Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=404 maxSegment=438
ts=2022-04-01T18:11:27.367Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=405 maxSegment=438
ts=2022-04-01T18:12:43.002Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=406 maxSegment=438
ts=2022-04-01T18:13:13.376Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=407 maxSegment=438
ts=2022-04-01T18:15:08.110Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=408 maxSegment=438
ts=2022-04-01T18:15:08.110Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=409 maxSegment=438
ts=2022-04-01T18:15:08.111Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=410 maxSegment=438
ts=2022-04-01T18:15:08.322Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..."
ts=2022-04-01T18:15:08.322Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..."
ts=2022-04-01T18:15:08.322Z caller=main.go:812 level=info msg="Stopping notify discovery manager..."
ts=2022-04-01T18:15:08.322Z caller=main.go:834 level=info msg="Stopping scrape manager..."
ts=2022-04-01T18:15:08.322Z caller=main.go:808 level=info msg="Notify discovery manager stopped"
ts=2022-04-01T18:15:08.322Z caller=main.go:828 level=info msg="Scrape manager stopped"
ts=2022-04-01T18:15:08.322Z caller=main.go:794 level=info msg="Scrape discovery manager stopped"
ts=2022-04-01T18:15:08.322Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-04-01T18:15:08.322Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-04-01T18:15:08.322Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-04-01T18:15:08.322Z caller=main.go:1054 level=info msg="Notifier manager stopped"
ts=2022-04-01T18:18:07.012Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=411 maxSegment=438
ts=2022-04-01T18:18:07.012Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=412 maxSegment=438
ts=2022-04-01T18:18:07.012Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=413 maxSegment=438
ts=2022-04-01T18:18:07.013Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=414 maxSegment=438
ts=2022-04-01T18:18:07.013Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=415 maxSegment=438
ts=2022-04-01T18:18:07.013Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=416 maxSegment=438
ts=2022-04-01T18:18:07.013Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=417 maxSegment=438
ts=2022-04-01T18:18:07.013Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=418 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=419 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=420 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=421 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=422 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=423 maxSegment=438
ts=2022-04-01T18:18:07.014Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=424 maxSegment=438
ts=2022-04-01T18:18:07.015Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=425 maxSegment=438
ts=2022-04-01T18:18:07.015Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=426 maxSegment=438
ts=2022-04-01T18:18:07.015Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=427 maxSegment=438
ts=2022-04-01T18:18:07.015Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=428 maxSegment=438
ts=2022-04-01T18:18:07.016Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=429 maxSegment=438
ts=2022-04-01T18:18:07.016Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=430 maxSegment=438
ts=2022-04-01T18:18:07.016Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=431 maxSegment=438
ts=2022-04-01T18:18:07.016Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=432 maxSegment=438
ts=2022-04-01T18:18:07.017Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=433 maxSegment=438
ts=2022-04-01T18:18:07.017Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=434 maxSegment=438
ts=2022-04-01T18:18:07.017Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=435 maxSegment=438
ts=2022-04-01T18:18:07.017Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=436 maxSegment=438
ts=2022-04-01T18:18:07.017Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=437 maxSegment=438
ts=2022-04-01T18:18:07.018Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=438 maxSegment=438
ts=2022-04-01T18:18:07.018Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=50.409542167s wal_replay_duration=17m55.057769237s total_replay_duration=18m47.379349668s
ts=2022-04-01T18:18:20.034Z caller=main.go:944 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-04-01T18:18:20.034Z caller=main.go:947 level=info msg="TSDB started"
ts=2022-04-01T18:18:20.034Z caller=main.go:1066 level=info msg="See you next time!"