Why Is Tf Build Failing with "Error Refreshing State: Http Remote State Endpoint Requires Auth"?

David Miller
David Miller
• • 7 min read

My Pipeline Builds, Plans and Applies Just Fine for My Dev Branch. When I Push to My Master Branch, I Get "Error Refreshing State: Http Remote State Endpoint...

Why Is Tf Build Failing with "Error Refreshing State: Http Remote State Endpoint Requires Auth"?

My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...

Running with gitlab-runner 13.4.1 (REDACTED)
  on runner-gitlab-runner-REDACTED-REDACTED REDACTED
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ...
Preparing environment
00:03
Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending
Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/
Created fresh repository.
Checking out REDACTED as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for REDACTEDREDACTEDREDACTEDREDACTED...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:05
$ git config --global credential.helper cache
$ git fetch
From 
 * [new branch]      dev        -> origin/dev
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
$ echo ${TF_ACCOUNT}
REDACTED
$ echo ${TF_ADDRESS}

$ echo TF_HTTP_ADDRESS=${TF_ADDRESS}
TF_HTTP_ADDRESS=
$ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
TF_HTTP_LOCK_ADDRESS=
$ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
TF_HTTP_UNLOCK_ADDRESS=
$ echo TF_ADDRESS=${TF_ADDRESS}
TF_ADDRESS=
$ export TF_HTTP_ADDRESS=${TF_ADDRESS}
$ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
$ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
$ export TF_HTTP_LOCK_METHOD="POST"
$ export TF_HTTP_UNLOCK_METHOD="DELETE"
$ export TF_HTTP_RETRY_WAIT_MIN='5'
$ export TF_HTTP_USERNAME='JOHN.DOE'
$ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
$ export TF_VAR_ACCOUNT=${TF_ACCOUNT}
$ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
$ export AWS_DEFAULT_REGION='us-east-1'
$ terraform init
Initializing modules...
Downloading git:: for lambda...
- lambda in .terraform/modules/lambda
Downloading git:: for lambda_iam...
- lambda_iam in .terraform/modules/lambda_iam
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint requires auth
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

A peek at my gitlab-ci.yml:

image:
  name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
stages:
  - build
  - plan
  - apply

variables:
  PLAN: plan.tfplan
  JSON_PLAN_FILE: tfplan.json
  WORKSPACE: "dev"
  TF_IN_AUTOMATION: "true"
  ZIP_FILE: "lambda_package.zip"
  STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
  TF_ACCOUNT: ""
  ORG_ACCOUNT: ""
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
  TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
  TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock

cache:
  key: "$CI_COMMIT_SHA"
  paths:
    - .terraform

before_script:
  - git config --global credential.helper cache 
  - git fetch 
  - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
  - echo ${TF_ACCOUNT}
  - echo ${TF_ADDRESS}
  - echo TF_HTTP_ADDRESS=${TF_ADDRESS}
  - echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
  - echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
  - echo TF_ADDRESS=${TF_ADDRESS}
  - export TF_HTTP_ADDRESS=${TF_ADDRESS}
  - export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
  - export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
  - export TF_HTTP_LOCK_METHOD="POST"
  - export TF_HTTP_UNLOCK_METHOD="DELETE"
  - export TF_HTTP_RETRY_WAIT_MIN='5'
  - export TF_HTTP_USERNAME='MATTHEW.FETHEROLF'
  - export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
  - export TF_VAR_ACCOUNT=${TF_ACCOUNT}
  - export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
  - export AWS_DEFAULT_REGION='us-east-1'
  - terraform init 

  

# -----BUILD-----
  # 1 build job per lambda function

buildLambda:
  stage: build
  tags:
    - cluster 
  script:
    - echo "Beginning Build"
    - cd lambda_code/
    - echo "Switched into Lambda dir"
    - pip3 install -r requirements.txt --target .
    - echo "Requirements installed"
    - echo $ZIP_FILE
    - zip -r $ZIP_FILE *
    - echo "Zip file packaged up"
  artifacts:
    paths:
      - lambda_code/
  only:
    - dev
    - master
    - merge_requests

# -----PLAN-----

planMerge:
  stage: plan
  tags:
  - cluster 
  script:
    - terraform plan
  dependencies:
    - buildLambda
  only:
    - merge_requests

planCommit:
  stage: plan
  tags:
    - cluster 
  script:
    - terraform plan
  dependencies:
    - buildLambda



# -----APPLY-----
  # dev branch will auto deploy
applyDev:
  stage: apply
  tags:
  - cluster 
  script:
    - echo "Running Terraform Apply"
    - terraform apply -auto-approve
  dependencies:
    - buildLambda
  only:
    - dev 
  #when: manual

    # prod branch will require manual deployment approval
applyProd:
  stage: apply
  tags:
  - cluster 
  script:
    - echo "Running Terraform Apply"
    - terraform apply -auto-approve
  when: manual
  dependencies:
    - buildLambda
  only:
    - master

backend.tf:

terraform {
    backend "http" {
    }
}

main.tf:

locals {
  common_tags = {
    SVC_ACCOUNT_ID = "REDACTED",
    CLOUD_PLATFORM_PROD_ID = "REDACTED"
  }
}

module "lambda" {
    source = "git::" 
    lambda_name = var.name
    lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}"
    lambda_handler = var.handler
    lambda_runtime = var.runtime
    default_lambda_timeout = var.timeout
    env = local.common_tags
    ACCOUNT = var.ACCOUNT
}

module "lambda_iam" {
    source = "git::" 
    lambda_policy = var.lambda_policy
    ACCOUNT = var.ACCOUNT
    lambda_role = var.lambda_role
}

inputs.tf:

variable "handler" {
  type = string
  default = "handler.lambda_handler"
}

variable "runtime" {
  type = string
  default = "python3.8"
}

variable "name" {
  type = string
  default = "onboard-cloud-account-into-monitoring"
}

variable "timeout"{
    type = string
    default = "120"
}

variable "lambda_role" {
  type = string
  default = "cloud-platform-onboard-to-monitoring"
}

variable "ACCOUNT" {
  type = string
}

variable "ORG_ACCOUNT" {
  type = string
}

variable "lambda_policy" {
  default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}"
}

provider.tf:


provider "aws" {
    region          = "us-east-1" 
    assume_role {
        role_arn    ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring"
    }

    default_tags {
            tags = {
                owner = "[email protected]"
                altowner = "[email protected]"
                blc = "REDACTED"
                costcenter = "REDACTED"
                itemid = "PlatformAutomation"
            }
        }
}

Perhaps I also need to share the terraform modules being invoked?

3

2 Answers

We found the problem: The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.

in my case there was no any protected environment variables but it was required to rename TF_STATE_NAME

Must Read

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

David Miller

David Miller

Executive Financial & Market Analyst
Twitter Linkedin Medium

David Miller brings 15 years of experience in global economics, personal finance strategy, and market dynamics. He specializes in turning complex economic trends into actionable insights for everyday readers.

Share this article
Twitter Facebook Pinterest