{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This post is a notebook. It contains instructions to set up and create a GitLab environment in Google Cloud Platform (GCP) using the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) (GET), installed into a [VS Code Dev Container](https://gitlab.com/mlockhart/get-container).\n",
    "\n",
    "The idea is that it's a _literate sysadmin_ runbook, which can be copied and repeated within my get-container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook instructions\n",
    "\n",
    "To use this, you must be running Jupyter Notebook with the Bash kernel installed.  This is [explored in detail on my earlier blog post](https://milosophical.me/blog/2025/vscode-jupyter-remote.html).  Jupyter and the extension should already have been installed when [the GET Dev Container](https://gitlab.com/mlockhart/get-container) was provisioned. If not, then re-run the playbook:\n",
    "\n",
    "```\n",
    "ansible-playbook $WORKSPACE/.devcontainer/get/setup.yaml\n",
    "```\n",
    "\n",
    "Then start jupyter:\n",
    "\n",
    "```\n",
    "jupyter notebook --no-browser --allow-root\n",
    "```\n",
    "(this is aliased to `notebook`)\n",
    "\n",
    "### Select Kernel\n",
    "\n",
    "Once jupyter is running in the dev container, you can open this notebook and select the **Bash** kernel from the `Select Kernel` button at the top of the buffer. The Bash kernel directly runs bash shell commands, rather than python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- TEASER_END -->\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate to GCP\n",
    "\n",
    "Each notebook starts with a fresh environment, so set it up first. This loads the shell functions and aliases — which are used throught the notebook — and authenticates to GCP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "source ~/.bashrc\n",
    "$WORKSPACE/.devcontainer/get/gcloud-auth.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "🛜  Connecting to the Googleplex\n",
    "\n",
    "\n",
    "GCP_AUTH_KIND:\tapplication\n",
    "GCP_PROJECT:\tmlockhart-56581c10\n",
    "GCP_REGION:\tus-west1\n",
    "GCP_SA_SSH:\t/workspaces/get/keys/service-account_rsa.pub\n",
    "GCP_SA_SSH_PRIV:\t/workspaces/get/keys/service-account_rsa\n",
    "GCP_SA_USER:\tsa_REDACTED\n",
    "GCP_TF_STATE_BUCKET:\tmjl-get-terraform-state\n",
    "GCP_ZONE:\tus-west1-c\n",
    "GET_ARCH:\t/workspaces/get/.devcontainer/arch\n",
    "GET_CONT:\t/workspaces/get/_CONFIG\n",
    "GET_HOME:\t/workspaces/get\n",
    "GET_KEYS:\t/workspaces/get/keys\n",
    "GET_LICENSE:\t/workspaces/get/licenses/gitlab-premium\n",
    "GET_ROOT_PASSWORD:\tdefaultpasswordchangeme\n",
    "GET_ULTIMATE:\t/workspaces/get/licenses/gitlab-ultimate\n",
    "GOOGLE_APPLICATION_CREDENTIALS:\t/workspaces/get/keys/mlockhart-56581c10-8bc0e30dd679.json\n",
    "GOOGLE_CREDENTIALS:\t/workspaces/get/keys/mlockhart-56581c10-8bc0e30dd679.json\n",
    "\n",
    "Updated property [core/project].\n",
    "Activated service account credentials for: [gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "loginProfile:\n",
    "    REDACTED\n",
    "    WARNING: Property validation for compute/region was skipped.\n",
    "Updated property [compute/region].\n",
    "WARNING: Property validation for compute/zone was skipped.\n",
    "Updated property [compute/zone].\n",
    "</code></pre>\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Reserve an external IP address\n",
    "\n",
    "The new environment must have an external IP address. I usually reserve one for each environment, and make the environment's prefix match the name of the IP address.\n",
    "\n",
    "It's good to list the current addresses that are reserved and in use, first. Here's the first shell alias. In this notebook I'll describe the aliases and functions so you can see what they mean. Normally you don't need to do that in order to just use them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "alias gaddress\n",
    "gaddress list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "\n",
    "alias gaddress='gcloud compute addresses'\n",
    "NAME                              ADDRESS/RANGE   TYPE      PURPOSE  NETWORK  REGION    SUBNET  STATUS\n",
    "gitlab-pages-ip                   104.198.10.38   EXTERNAL                    us-west1          RESERVED\n",
    "gitlab-suse                       34.83.25.79     EXTERNAL                    us-west1          IN_USE\n",
    "gl0                               34.19.85.189    EXTERNAL                    us-west1          RESERVED\n",
    "gl1                               35.247.117.44   EXTERNAL                    us-west1          IN_USE\n",
    "gl2                               34.127.19.111   EXTERNAL                    us-west1          IN_USE\n",
    "gl3                               34.127.83.216   EXTERNAL                    us-west1          RESERVED\n",
    "gl4                               34.83.247.87    EXTERNAL                    us-west1          RESERVED\n",
    "gl4u                              34.83.115.41    EXTERNAL                    us-west1          RESERVED\n",
    "gl5                               34.169.19.81    EXTERNAL                    us-west1          IN_USE\n",
    "gl6-a                             34.168.235.194  EXTERNAL                    us-west1          IN_USE\n",
    "gl6-s                             34.105.121.54   EXTERNAL                    us-west1          IN_USE\n",
    "gl7                               34.145.108.180  EXTERNAL                    us-west1          RESERVED\n",
    "interview-static-ip               34.169.52.156   EXTERNAL                    us-west1          RESERVED\n",
    "mjl5658-gl2-2k-gitaly-ip-1        35.197.37.23    EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-gitlab-rails-ip-1  34.19.3.76      EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-gitlab-rails-ip-2  34.105.50.40    EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-monitor-ip-1       34.82.163.222   EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-postgres-ip-1      104.198.109.85  EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-redis-ip-1         34.168.118.24   EXTERNAL                    us-west1          IN_USE\n",
    "mjl5658-gl2-2k-sidekiq-ip-1       34.82.105.251   EXTERNAL                    us-west1          IN_USE\n",
    "new-p                             35.197.120.186  EXTERNAL                    us-west1          IN_USE\n",
    "new-s                             34.82.12.174    EXTERNAL                    us-west1          IN_USE\n",
    "old-p                             34.105.64.192   EXTERNAL                    us-west1          RESERVED\n",
    "runner-static-ip                  35.199.147.59   EXTERNAL                    us-west1          IN_USE\n",
    "\n",
    "</code></pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Create an address if needed\n",
    "\n",
    "Reserve a new address if necessary. Give it a short name that will be used for the environment. Here I create a new `gl8` address:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created [https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/regions/us-west1/addresses/gl8].\n"
     ]
    }
   ],
   "source": [
    "gaddress create gl8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command does not report what the new address is. We can retrieve the generated IP address using another function:\n",
    "\n",
    "```\n",
    "type gaddress-ip\n",
    "```\n",
    "```\n",
    "gaddress-ip is a function\n",
    "gaddress-ip () \n",
    "{ \n",
    "    local FUNCDESC='Print the IP addresses for matching GCP server names';\n",
    "    if [[ -z ${1} ]]; then\n",
    "        echo \"Usage: ${FUNCNAME} «name_pattern»\" ${FUNCDESC};\n",
    "        return 1;\n",
    "    fi;\n",
    "    gcloud compute addresses list --filter=\"name~'${1}'\" --format=\"value(address)\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Generate a GET configuration\n",
    "\n",
    "Use the [gcp-environment](https://gitlab.com/mlockhart/get-container/-/blob/main/bin/gcp-environment.py) script included in _get-container_, to create a new enironment from the GET configuration templates.\n",
    "\n",
    "Below are the required script arguments, in long form. You can also run `gcp-environment --help` to see all of the options. The `--economic` option will configure Terraform to use less powerful compute instances with smaller volumes, which is fine and saves cost for a single-engineer test environment, but will not meet the performance specs for a reference architecture.\n",
    "\n",
    "Change the values as necessary, and then run the cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(prefix='gl8', external_ip='34.168.4.189', url=None, arch='1k', glver='17.4.1', password='ur5iY5Cea8yez9Oo', license='/workspaces/get/licenses/gitlab-premium', economic=True, disk_size=20, ident='mjl5658', project='mlockhart-56581c10', region='us-west1', zone='us-west1-c', bucket='mjl-get-terraform-state', ssh_username='sa_104481838115385290889', ssh_key_path='/workspaces/get/keys/service-account_rsa', ansible='/workspaces/get/ansible/environments', terraform='/workspaces/get/terraform/environments', templates='/workspaces/get/.devcontainer/arch')\n",
      "Generated /workspaces/get/terraform/environments/gl8_1k_v170401/environment.tf\n",
      "Generated /workspaces/get/terraform/environments/gl8_1k_v170401/main.tf\n",
      "Generated /workspaces/get/terraform/environments/gl8_1k_v170401/variables.tf\n",
      "Generated /workspaces/get/ansible/environments/gl8_1k_v170401/inventory/1k.gcp.yml\n",
      "Generated /workspaces/get/ansible/environments/gl8_1k_v170401/inventory/vars.yml\n"
     ]
    }
   ],
   "source": [
    "cd $WORKSPACE\n",
    "gcp-environment \\\n",
    "  --prefix gl8 \\\n",
    "  --external_ip $(gaddress-ip gl8) \\\n",
    "  --arch 1k \\\n",
    "  --glver 17.4.1 \\\n",
    "  --password $(pwgen 16) \\\n",
    "  --economic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script created new directories in GET's `terraform/environments` and `ansible/environments`, and added the necessary GET configuration settings.\n",
    "\n",
    "You can inspect and fine-tune the settings in these files before applying them in Terraform and Ansible. If you are viewing this notebook in VSCode, you can click on the files above to open them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Terraform\n",
    "\n",
    "This step creates the necessary infrastructure in GCP for the new GitLab environment. It uses the Terraform files we just created.\n",
    "\n",
    "The terraform settings need to be run from the terraform environment directory. Here are shortcuts to get there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alias cdte='cd ${GET_HOME}/terraform/environments/'\n"
     ]
    }
   ],
   "source": [
    "alias cdte\n",
    "cdte"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're in the terraform environment base directory for GET. Switch to the specific environment's directory next. Doing it this way in two steps facilitates <kbd>TAB</kbd>-key completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspaces/get/terraform/environments/gl8_1k_v170401\n"
     ]
    }
   ],
   "source": [
    "cd gl8_1k_v170401"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Terraform init\n",
    "\n",
    "Terraform must be initialized before we can plan the infrastructure changes, or apply them. Use another alias for `terraform`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "alias tf\n",
    "tf init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "\n",
    "alias tf='time terraform'\n",
    "Initializing the backend...\n",
    "\n",
    "Successfully configured the backend \"gcs\"! Terraform will automatically\n",
    "use this backend unless the backend configuration changes.\n",
    "Initializing modules...\n",
    "- gitlab_ref_arch_gcp in ../../modules/gitlab_ref_arch_gcp\n",
    "- gitlab_ref_arch_gcp.consul in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.consul.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitaly in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.gitaly.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_gitaly_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_node_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_registry_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_sidekiq_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_supporting_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_toolbox_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_gke_webservice_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_nfs in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.gitlab_nfs.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.gitlab_rails in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.gitlab_rails.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.haproxy_external in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.haproxy_external.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.haproxy_internal in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.haproxy_internal.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.monitor in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.monitor.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.opensearch_vm in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.opensearch_vm.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.pgbouncer in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.pgbouncer.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.postgres in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.postgres.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.praefect in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.praefect.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.praefect_postgres in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.praefect_postgres.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.redis in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.redis.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.redis_cache in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.redis_cache.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.redis_persistent in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.redis_persistent.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "- gitlab_ref_arch_gcp.sidekiq in ../../modules/gitlab_gcp_instance\n",
    "- gitlab_ref_arch_gcp.sidekiq.google_service_account in ../../modules/gitlab_gcp_service_account\n",
    "Initializing provider plugins...\n",
    "- Finding hashicorp/google versions matching \"~> 6.0\"...\n",
    "- Installing hashicorp/google v6.30.0...\n",
    "- Installed hashicorp/google v6.30.0 (signed by HashiCorp)\n",
    "Terraform has created a lock file .terraform.lock.hcl to record the provider\n",
    "selections it made above. Include this file in your version control repository\n",
    "so that Terraform can guarantee to make the same selections by default when\n",
    "you run \"terraform init\" in the future.\n",
    "\n",
    "Terraform has been successfully initialized!\n",
    "\n",
    "You may now begin working with Terraform. Try running \"terraform plan\" to see\n",
    "any changes that are required for your infrastructure. All Terraform commands\n",
    "should now work.\n",
    "\n",
    "If you ever set or change modules or backend configuration for Terraform,\n",
    "rerun this command to reinitialize your working directory. If you forget, other\n",
    "commands will detect it and remind you to do so if necessary.\n",
    "\n",
    "real\t0m11.086s\n",
    "user\t0m0.861s\n",
    "sys\t0m0.482s\n",
    "\n",
    "</code></pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Terraform apply\n",
    "\n",
    "You _should_ plan first.`¯\\_(ツ)_/¯` You can do that with the `plan` action:\n",
    "\n",
    "```\n",
    "tf plan\n",
    "```\n",
    "\n",
    "For testing, I usually skip the plan.\n",
    "\n",
    "Here is another alias, to automatically approve and apply the terraform plan:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "alias tfap\n",
    "tfap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "\n",
    "alias tfap='time terraform apply --auto-approve=true'\n",
    "Acquiring state lock. This may take a few moments...\n",
    "module.gitlab_ref_arch_gcp.data.google_compute_network.gitlab_network[0]: Reading...\n",
    "module.gitlab_ref_arch_gcp.data.google_compute_subnetwork.gitlab_subnet[0]: Reading...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].data.google_client_openid_userinfo.current_userinfo[0]: Reading...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].data.google_client_openid_userinfo.current_userinfo[0]: Read complete after 1s [id=gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.data.google_compute_network.gitlab_network[0]: Read complete after 1s [id=projects/mlockhart-56581c10/global/networks/default]\n",
    "module.gitlab_ref_arch_gcp.data.google_compute_subnetwork.gitlab_subnet[0]: Read complete after 1s [id=projects/mlockhart-56581c10/regions/us-west1/subnetworks/default]\n",
    "\n",
    "Terraform used the selected providers to generate the following execution plan.\n",
    "Resource actions are indicated with the following symbols:\n",
    "  + create\n",
    "\n",
    "Terraform will perform the following actions:\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_http_https_access[0] will be created\n",
    "  + resource \"google_compute_firewall\" \"gitlab_http_https_access\" {\n",
    "      + creation_timestamp = (known after apply)\n",
    "      + description        = \"Enable HTTP / HTTPS access for GitLab environment 'mjl5658-gl8-1k-v170401' on the default network\"\n",
    "      + destination_ranges = (known after apply)\n",
    "      + direction          = (known after apply)\n",
    "      + enable_logging     = (known after apply)\n",
    "      + id                 = (known after apply)\n",
    "      + name               = \"mjl5658-gl8-1k-v170401-http-https-access\"\n",
    "      + network            = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "      + priority           = 1000\n",
    "      + project            = \"mlockhart-56581c10\"\n",
    "      + self_link          = (known after apply)\n",
    "      + source_ranges      = [\n",
    "          + \"0.0.0.0/0\",\n",
    "        ]\n",
    "      + target_tags        = [\n",
    "          + \"mjl5658-gl8-1k-v170401-web\",\n",
    "        ]\n",
    "\n",
    "      + allow {\n",
    "          + ports    = [\n",
    "              + \"80\",\n",
    "              + \"443\",\n",
    "            ]\n",
    "          + protocol = \"tcp\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_icmp_access[0] will be created\n",
    "  + resource \"google_compute_firewall\" \"gitlab_icmp_access\" {\n",
    "      + creation_timestamp = (known after apply)\n",
    "      + description        = \"Enable ICMP access for GitLab environment 'mjl5658-gl8-1k-v170401' on the default network\"\n",
    "      + destination_ranges = (known after apply)\n",
    "      + direction          = (known after apply)\n",
    "      + enable_logging     = (known after apply)\n",
    "      + id                 = (known after apply)\n",
    "      + name               = \"mjl5658-gl8-1k-v170401-icmp-access\"\n",
    "      + network            = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "      + priority           = 1000\n",
    "      + project            = \"mlockhart-56581c10\"\n",
    "      + self_link          = (known after apply)\n",
    "      + source_ranges      = [\n",
    "          + \"0.0.0.0/0\",\n",
    "        ]\n",
    "      + target_tags        = [\n",
    "          + \"mjl5658-gl8-1k-v170401\",\n",
    "        ]\n",
    "\n",
    "      + allow {\n",
    "          + ports    = []\n",
    "          + protocol = \"icmp\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_vm_ssh_access[0] will be created\n",
    "  + resource \"google_compute_firewall\" \"gitlab_vm_ssh_access\" {\n",
    "      + creation_timestamp = (known after apply)\n",
    "      + description        = \"Enable SSH access to VMs for GitLab environment 'mjl5658-gl8-1k-v170401' on the default network\"\n",
    "      + destination_ranges = (known after apply)\n",
    "      + direction          = (known after apply)\n",
    "      + enable_logging     = (known after apply)\n",
    "      + id                 = (known after apply)\n",
    "      + name               = \"mjl5658-gl8-1k-v170401-vm-ssh-access\"\n",
    "      + network            = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "      + priority           = 1000\n",
    "      + project            = \"mlockhart-56581c10\"\n",
    "      + self_link          = (known after apply)\n",
    "      + source_ranges      = [\n",
    "          + \"0.0.0.0/0\",\n",
    "        ]\n",
    "      + target_tags        = [\n",
    "          + \"mjl5658-gl8-1k-v170401\",\n",
    "        ]\n",
    "\n",
    "      + allow {\n",
    "          + ports    = [\n",
    "              + \"22\",\n",
    "            ]\n",
    "          + protocol = \"tcp\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"artifacts\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-artifacts\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"backups\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-backups\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"ci-secure-files\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-ci-secure-files\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"dependency-proxy\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-dependency-proxy\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"lfs\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-lfs\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"mr-diffs\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-mr-diffs\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"packages\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-packages\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"pages\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-pages\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"registry\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-registry\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"terraform-state\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-terraform-state\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"uploads\"] will be created\n",
    "  + resource \"google_storage_bucket\" \"gitlab_object_storage_buckets\" {\n",
    "      + effective_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + force_destroy               = true\n",
    "      + id                          = (known after apply)\n",
    "      + location                    = \"US\"\n",
    "      + name                        = \"mjl5658-gl8-1k-v170401-uploads\"\n",
    "      + project                     = (known after apply)\n",
    "      + project_number              = (known after apply)\n",
    "      + public_access_prevention    = \"enforced\"\n",
    "      + rpo                         = (known after apply)\n",
    "      + self_link                   = (known after apply)\n",
    "      + storage_class               = \"STANDARD\"\n",
    "      + terraform_labels            = {\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + uniform_bucket_level_access = true\n",
    "      + url                         = (known after apply)\n",
    "\n",
    "      + soft_delete_policy (known after apply)\n",
    "\n",
    "      + versioning {\n",
    "          + enabled = false\n",
    "        }\n",
    "\n",
    "      + website (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"artifacts\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-artifacts\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"backups\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-backups\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"ci-secure-files\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-ci-secure-files\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"dependency-proxy\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-dependency-proxy\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"lfs\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-lfs\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"mr-diffs\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-mr-diffs\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"packages\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-packages\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"pages\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-pages\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"registry\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-registry\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"terraform-state\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-terraform-state\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"uploads\"] will be created\n",
    "  + resource \"google_storage_bucket_iam_member\" \"gitlab_rails_object_storage_buckets_member\" {\n",
    "      + bucket = \"mjl5658-gl8-1k-v170401-uploads\"\n",
    "      + etag   = (known after apply)\n",
    "      + id     = (known after apply)\n",
    "      + member = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role   = \"roles/storage.objectAdmin\"\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.module.gitlab_rails.google_compute_instance.gitlab[0] will be created\n",
    "  + resource \"google_compute_instance\" \"gitlab\" {\n",
    "      + allow_stopping_for_update = true\n",
    "      + can_ip_forward            = false\n",
    "      + cpu_platform              = (known after apply)\n",
    "      + creation_timestamp        = (known after apply)\n",
    "      + current_status            = (known after apply)\n",
    "      + deletion_protection       = false\n",
    "      + effective_labels          = {\n",
    "          + \"gitlab_node_level\"          = \"gitlab-rails-primary\"\n",
    "          + \"gitlab_node_prefix\"         = \"mjl5658-gl8-1k-v170401\"\n",
    "          + \"gitlab_node_type\"           = \"gitlab-rails\"\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + id                        = (known after apply)\n",
    "      + instance_id               = (known after apply)\n",
    "      + label_fingerprint         = (known after apply)\n",
    "      + labels                    = {\n",
    "          + \"gitlab_node_level\"  = \"gitlab-rails-primary\"\n",
    "          + \"gitlab_node_prefix\" = \"mjl5658-gl8-1k-v170401\"\n",
    "          + \"gitlab_node_type\"   = \"gitlab-rails\"\n",
    "        }\n",
    "      + machine_type              = \"e2-standard-2\"\n",
    "      + metadata                  = {\n",
    "          + \"enable-oslogin\" = \"TRUE\"\n",
    "        }\n",
    "      + metadata_fingerprint      = (known after apply)\n",
    "      + min_cpu_platform          = (known after apply)\n",
    "      + name                      = \"mjl5658-gl8-1k-v170401-gitlab-rails-1\"\n",
    "      + project                   = \"mlockhart-56581c10\"\n",
    "      + self_link                 = (known after apply)\n",
    "      + tags                      = [\n",
    "          + \"gitlab-rails\",\n",
    "          + \"mjl5658-gl8-1k-v170401\",\n",
    "          + \"mjl5658-gl8-1k-v170401-gitlab-rails\",\n",
    "          + \"mjl5658-gl8-1k-v170401-ssh\",\n",
    "          + \"mjl5658-gl8-1k-v170401-web\",\n",
    "        ]\n",
    "      + tags_fingerprint          = (known after apply)\n",
    "      + terraform_labels          = {\n",
    "          + \"gitlab_node_level\"          = \"gitlab-rails-primary\"\n",
    "          + \"gitlab_node_prefix\"         = \"mjl5658-gl8-1k-v170401\"\n",
    "          + \"gitlab_node_type\"           = \"gitlab-rails\"\n",
    "          + \"goog-terraform-provisioned\" = \"true\"\n",
    "        }\n",
    "      + zone                      = \"us-west1-c\"\n",
    "\n",
    "      + boot_disk {\n",
    "          + auto_delete                = true\n",
    "          + device_name                = (known after apply)\n",
    "          + disk_encryption_key_sha256 = (known after apply)\n",
    "          + guest_os_features          = (known after apply)\n",
    "          + kms_key_self_link          = (known after apply)\n",
    "          + mode                       = \"READ_WRITE\"\n",
    "          + source                     = (known after apply)\n",
    "\n",
    "          + initialize_params {\n",
    "              + architecture           = (known after apply)\n",
    "              + image                  = \"ubuntu-2404-lts-amd64\"\n",
    "              + labels                 = (known after apply)\n",
    "              + provisioned_iops       = (known after apply)\n",
    "              + provisioned_throughput = (known after apply)\n",
    "              + resource_policies      = (known after apply)\n",
    "              + size                   = 20\n",
    "              + snapshot               = (known after apply)\n",
    "              + type                   = \"pd-standard\"\n",
    "            }\n",
    "        }\n",
    "\n",
    "      + confidential_instance_config (known after apply)\n",
    "\n",
    "      + guest_accelerator (known after apply)\n",
    "\n",
    "      + network_interface {\n",
    "          + internal_ipv6_prefix_length = (known after apply)\n",
    "          + ipv6_access_type            = (known after apply)\n",
    "          + ipv6_address                = (known after apply)\n",
    "          + name                        = (known after apply)\n",
    "          + network                     = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "          + network_attachment          = (known after apply)\n",
    "          + network_ip                  = (known after apply)\n",
    "          + stack_type                  = (known after apply)\n",
    "          + subnetwork                  = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/regions/us-west1/subnetworks/default\"\n",
    "          + subnetwork_project          = (known after apply)\n",
    "\n",
    "          + access_config {\n",
    "              + nat_ip       = \"34.168.4.189\"\n",
    "              + network_tier = (known after apply)\n",
    "            }\n",
    "        }\n",
    "\n",
    "      + reservation_affinity (known after apply)\n",
    "\n",
    "      + scheduling (known after apply)\n",
    "\n",
    "      + service_account {\n",
    "          + email  = \"gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "          + scopes = [\n",
    "              + \"https://www.googleapis.com/auth/cloud-platform\",\n",
    "            ]\n",
    "        }\n",
    "\n",
    "      + shielded_instance_config {\n",
    "          + enable_integrity_monitoring = true\n",
    "          + enable_secure_boot          = false\n",
    "          + enable_vtpm                 = true\n",
    "        }\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account.gitlab[0] will be created\n",
    "  + resource \"google_service_account\" \"gitlab\" {\n",
    "      + account_id   = \"gl8-gitlab-rails\"\n",
    "      + disabled     = false\n",
    "      + display_name = \"mjl5658-gl8-1k-v170401-gitlab-rails\"\n",
    "      + email        = \"gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + id           = (known after apply)\n",
    "      + member       = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + name         = (known after apply)\n",
    "      + project      = \"mlockhart-56581c10\"\n",
    "      + unique_id    = (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.gitlab_user_members[\"serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com\"] will be created\n",
    "  + resource \"google_service_account_iam_member\" \"gitlab_user_members\" {\n",
    "      + etag               = (known after apply)\n",
    "      + id                 = (known after apply)\n",
    "      + member             = \"serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role               = \"roles/iam.serviceAccountUser\"\n",
    "      + service_account_id = (known after apply)\n",
    "    }\n",
    "\n",
    "  # module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.object_storage_profile_token_creator[0] will be created\n",
    "  + resource \"google_service_account_iam_member\" \"object_storage_profile_token_creator\" {\n",
    "      + etag               = (known after apply)\n",
    "      + id                 = (known after apply)\n",
    "      + member             = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      + role               = \"roles/iam.serviceAccountTokenCreator\"\n",
    "      + service_account_id = (known after apply)\n",
    "    }\n",
    "\n",
    "Plan: 29 to add, 0 to change, 0 to destroy.\n",
    "\n",
    "Changes to Outputs:\n",
    "  + gitlab_ref_arch_gcp = {\n",
    "      + cloud_sql_geo_tracking_postgres_connection = {\n",
    "          + cloud_sql_host    = \"\"\n",
    "          + cloud_sql_name    = \"\"\n",
    "          + cloud_sql_version = \"\"\n",
    "        }\n",
    "      + cloud_sql_postgres_connection              = {\n",
    "          + cloud_sql_host               = \"\"\n",
    "          + cloud_sql_name               = \"\"\n",
    "          + cloud_sql_read_replica_hosts = []\n",
    "          + cloud_sql_version            = \"\"\n",
    "        }\n",
    "      + cloud_sql_praefect_postgres_connection     = {\n",
    "          + cloud_sql_host    = \"\"\n",
    "          + cloud_sql_name    = \"\"\n",
    "          + cloud_sql_version = \"\"\n",
    "        }\n",
    "      + consul                                     = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + gitaly                                     = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + gitlab_nfs                                 = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + gitlab_rails                               = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = [\n",
    "              + \"34.168.4.189\",\n",
    "            ]\n",
    "          + internal_addresses     = {\n",
    "              + \"mjl5658-gl8-1k-v170401-gitlab-rails-1.c.mlockhart-56581c10.internal\" = (known after apply)\n",
    "            }\n",
    "          + machine_names          = [\n",
    "              + \"mjl5658-gl8-1k-v170401-gitlab-rails-1\",\n",
    "            ]\n",
    "          + self_links             = [\n",
    "              + (known after apply),\n",
    "            ]\n",
    "          + service_account        = {\n",
    "              + email     = \"gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "              + member    = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "              + name      = (known after apply)\n",
    "              + unique_id = (known after apply)\n",
    "            }\n",
    "          + zones                  = [\n",
    "              + \"us-west1-c\",\n",
    "            ]\n",
    "        }\n",
    "      + haproxy_external                           = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + haproxy_internal                           = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + kubernetes                                 = {\n",
    "          + kubernetes_cluster_id                                  = \"\"\n",
    "          + kubernetes_cluster_name                                = \"\"\n",
    "          + kubernetes_cluster_version                             = \"\"\n",
    "          + kubernetes_gitaly_node_group_service_account_email     = \"\"\n",
    "          + kubernetes_gitaly_node_group_version                   = \"\"\n",
    "          + kubernetes_node_group_service_account_email            = \"\"\n",
    "          + kubernetes_registry_service_account_email              = \"\"\n",
    "          + kubernetes_sidekiq_node_group_service_account_email    = \"\"\n",
    "          + kubernetes_sidekiq_node_group_version                  = \"\"\n",
    "          + kubernetes_sidekiq_service_account_email               = \"\"\n",
    "          + kubernetes_supporting_node_group_service_account_email = \"\"\n",
    "          + kubernetes_supporting_node_group_version               = \"\"\n",
    "          + kubernetes_toolbox_service_account_email               = \"\"\n",
    "          + kubernetes_webservice_node_group_service_account_email = \"\"\n",
    "          + kubernetes_webservice_node_group_version               = \"\"\n",
    "          + kubernetes_webservice_service_account_email            = \"\"\n",
    "        }\n",
    "      + memorystore_redis_cache_connection         = {\n",
    "          + memorystore_redis_cache_address = \"\"\n",
    "          + memorystore_redis_cache_port    = \"\"\n",
    "        }\n",
    "      + memorystore_redis_connection               = {\n",
    "          + memorystore_redis_address = \"\"\n",
    "          + memorystore_redis_port    = \"\"\n",
    "        }\n",
    "      + memorystore_redis_persistent_connection    = {\n",
    "          + memorystore_redis_persistent_address = \"\"\n",
    "          + memorystore_redis_persistent_port    = \"\"\n",
    "        }\n",
    "      + monitor                                    = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + network                                    = {\n",
    "          + vpc_id                = \"projects/mlockhart-56581c10/global/networks/default\"\n",
    "          + vpc_name              = \"default\"\n",
    "          + vpc_self_link         = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "          + vpc_subnet_cidr_block = \"10.138.0.0/20\"\n",
    "          + vpc_subnet_id         = \"projects/mlockhart-56581c10/regions/us-west1/subnetworks/default\"\n",
    "          + vpc_subnet_name       = \"default\"\n",
    "          + vpc_subnet_self_link  = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/regions/us-west1/subnetworks/default\"\n",
    "        }\n",
    "      + opensearch_vm                              = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + pgbouncer                                  = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + postgres                                   = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + praefect                                   = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + praefect_postgres                          = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + redis                                      = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + redis_cache                                = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + redis_persistent                           = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "      + sidekiq                                    = {\n",
    "          + data_disk_device_names = []\n",
    "          + external_addresses     = []\n",
    "          + internal_addresses     = {}\n",
    "          + machine_names          = []\n",
    "          + self_links             = []\n",
    "          + service_account        = null\n",
    "          + zones                  = []\n",
    "        }\n",
    "    }\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"artifacts\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"ci-secure-files\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account.gitlab[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"uploads\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"terraform-state\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"packages\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"lfs\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"dependency-proxy\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"mr-diffs\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"pages\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"lfs\"]: Creation complete after 4s [id=mjl5658-gl8-1k-v170401-lfs]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"backups\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"packages\"]: Creation complete after 4s [id=mjl5658-gl8-1k-v170401-packages]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"registry\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"dependency-proxy\"]: Creation complete after 5s [id=mjl5658-gl8-1k-v170401-dependency-proxy]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_vm_ssh_access[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"artifacts\"]: Creation complete after 6s [id=mjl5658-gl8-1k-v170401-artifacts]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_http_https_access[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"mr-diffs\"]: Creation complete after 6s [id=mjl5658-gl8-1k-v170401-mr-diffs]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_icmp_access[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"backups\"]: Creation complete after 3s [id=mjl5658-gl8-1k-v170401-backups]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"terraform-state\"]: Creation complete after 7s [id=mjl5658-gl8-1k-v170401-terraform-state]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"uploads\"]: Creation complete after 7s [id=mjl5658-gl8-1k-v170401-uploads]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"ci-secure-files\"]: Creation complete after 7s [id=mjl5658-gl8-1k-v170401-ci-secure-files]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"pages\"]: Creation complete after 8s [id=mjl5658-gl8-1k-v170401-pages]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket.gitlab_object_storage_buckets[\"registry\"]: Creation complete after 4s [id=mjl5658-gl8-1k-v170401-registry]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account.gitlab[0]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account.gitlab[0]: Creation complete after 15s [id=projects/mlockhart-56581c10/serviceAccounts/gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.object_storage_profile_token_creator[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.gitlab_user_members[\"serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"ci-secure-files\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"backups\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"uploads\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"pages\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.google_compute_instance.gitlab[0]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_vm_ssh_access[0]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_http_https_access[0]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_icmp_access[0]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_vm_ssh_access[0]: Creation complete after 13s [id=projects/mlockhart-56581c10/global/firewalls/mjl5658-gl8-1k-v170401-vm-ssh-access]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"packages\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_http_https_access[0]: Creation complete after 12s [id=projects/mlockhart-56581c10/global/firewalls/mjl5658-gl8-1k-v170401-http-https-access]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"mr-diffs\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_compute_firewall.gitlab_icmp_access[0]: Creation complete after 12s [id=projects/mlockhart-56581c10/global/firewalls/mjl5658-gl8-1k-v170401-icmp-access]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"artifacts\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"backups\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-backups/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"pages\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-pages/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"terraform-state\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"dependency-proxy\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.object_storage_profile_token_creator[0]: Creation complete after 5s [id=projects/mlockhart-56581c10/serviceAccounts/gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com/roles/iam.serviceAccountTokenCreator/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"ci-secure-files\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-ci-secure-files/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"uploads\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-uploads/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"registry\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"lfs\"]: Creating...\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"packages\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-packages/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"mr-diffs\"]: Creation complete after 6s [id=b/mjl5658-gl8-1k-v170401-mr-diffs/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"artifacts\"]: Creation complete after 6s [id=b/mjl5658-gl8-1k-v170401-artifacts/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.gitlab_user_members[\"serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com\"]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.google_compute_instance.gitlab[0]: Still creating... [10s elapsed]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"dependency-proxy\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-dependency-proxy/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.module.google_service_account[0].google_service_account_iam_member.gitlab_user_members[\"serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com\"]: Creation complete after 10s [id=projects/mlockhart-56581c10/serviceAccounts/gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:gitlab-mjl-mac@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"terraform-state\"]: Creation complete after 5s [id=b/mjl5658-gl8-1k-v170401-terraform-state/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"registry\"]: Creation complete after 6s [id=b/mjl5658-gl8-1k-v170401-registry/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.google_storage_bucket_iam_member.gitlab_rails_object_storage_buckets_member[\"lfs\"]: Creation complete after 6s [id=b/mjl5658-gl8-1k-v170401-lfs/roles/storage.objectAdmin/serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.google_compute_instance.gitlab[0]: Still creating... [20s elapsed]\n",
    "module.gitlab_ref_arch_gcp.module.gitlab_rails.google_compute_instance.gitlab[0]: Creation complete after 23s [id=projects/mlockhart-56581c10/zones/us-west1-c/instances/mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "Apply complete! Resources: 29 added, 0 changed, 0 destroyed.\n",
    "\n",
    "Outputs:\n",
    "\n",
    "gitlab_ref_arch_gcp = {\n",
    "  \"cloud_sql_geo_tracking_postgres_connection\" = {\n",
    "    \"cloud_sql_host\" = \"\"\n",
    "    \"cloud_sql_name\" = \"\"\n",
    "    \"cloud_sql_version\" = \"\"\n",
    "  }\n",
    "  \"cloud_sql_postgres_connection\" = {\n",
    "    \"cloud_sql_host\" = \"\"\n",
    "    \"cloud_sql_name\" = \"\"\n",
    "    \"cloud_sql_read_replica_hosts\" = []\n",
    "    \"cloud_sql_version\" = \"\"\n",
    "  }\n",
    "  \"cloud_sql_praefect_postgres_connection\" = {\n",
    "    \"cloud_sql_host\" = \"\"\n",
    "    \"cloud_sql_name\" = \"\"\n",
    "    \"cloud_sql_version\" = \"\"\n",
    "  }\n",
    "  \"consul\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"gitaly\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"gitlab_nfs\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"gitlab_rails\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = tolist([\n",
    "      \"34.168.4.189\",\n",
    "    ])\n",
    "    \"internal_addresses\" = {\n",
    "      \"mjl5658-gl8-1k-v170401-gitlab-rails-1.c.mlockhart-56581c10.internal\" = \"10.138.0.78\"\n",
    "    }\n",
    "    \"machine_names\" = [\n",
    "      \"mjl5658-gl8-1k-v170401-gitlab-rails-1\",\n",
    "    ]\n",
    "    \"self_links\" = [\n",
    "      \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/zones/us-west1-c/instances/mjl5658-gl8-1k-v170401-gitlab-rails-1\",\n",
    "    ]\n",
    "    \"service_account\" = {\n",
    "      \"email\" = \"gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      \"member\" = \"serviceAccount:gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      \"name\" = \"projects/mlockhart-56581c10/serviceAccounts/gl8-gitlab-rails@mlockhart-56581c10.iam.gserviceaccount.com\"\n",
    "      \"unique_id\" = \"111971693028345180408\"\n",
    "    }\n",
    "    \"zones\" = [\n",
    "      \"us-west1-c\",\n",
    "    ]\n",
    "  }\n",
    "  \"haproxy_external\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"haproxy_internal\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"kubernetes\" = {\n",
    "    \"kubernetes_cluster_id\" = \"\"\n",
    "    \"kubernetes_cluster_name\" = \"\"\n",
    "    \"kubernetes_cluster_version\" = \"\"\n",
    "    \"kubernetes_gitaly_node_group_service_account_email\" = \"\"\n",
    "    \"kubernetes_gitaly_node_group_version\" = \"\"\n",
    "    \"kubernetes_node_group_service_account_email\" = \"\"\n",
    "    \"kubernetes_registry_service_account_email\" = \"\"\n",
    "    \"kubernetes_sidekiq_node_group_service_account_email\" = \"\"\n",
    "    \"kubernetes_sidekiq_node_group_version\" = \"\"\n",
    "    \"kubernetes_sidekiq_service_account_email\" = \"\"\n",
    "    \"kubernetes_supporting_node_group_service_account_email\" = \"\"\n",
    "    \"kubernetes_supporting_node_group_version\" = \"\"\n",
    "    \"kubernetes_toolbox_service_account_email\" = \"\"\n",
    "    \"kubernetes_webservice_node_group_service_account_email\" = \"\"\n",
    "    \"kubernetes_webservice_node_group_version\" = \"\"\n",
    "    \"kubernetes_webservice_service_account_email\" = \"\"\n",
    "  }\n",
    "  \"memorystore_redis_cache_connection\" = {\n",
    "    \"memorystore_redis_cache_address\" = \"\"\n",
    "    \"memorystore_redis_cache_port\" = \"\"\n",
    "  }\n",
    "  \"memorystore_redis_connection\" = {\n",
    "    \"memorystore_redis_address\" = \"\"\n",
    "    \"memorystore_redis_port\" = \"\"\n",
    "  }\n",
    "  \"memorystore_redis_persistent_connection\" = {\n",
    "    \"memorystore_redis_persistent_address\" = \"\"\n",
    "    \"memorystore_redis_persistent_port\" = \"\"\n",
    "  }\n",
    "  \"monitor\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"network\" = {\n",
    "    \"vpc_id\" = \"projects/mlockhart-56581c10/global/networks/default\"\n",
    "    \"vpc_name\" = \"default\"\n",
    "    \"vpc_self_link\" = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/global/networks/default\"\n",
    "    \"vpc_subnet_cidr_block\" = \"10.138.0.0/20\"\n",
    "    \"vpc_subnet_id\" = \"projects/mlockhart-56581c10/regions/us-west1/subnetworks/default\"\n",
    "    \"vpc_subnet_name\" = \"default\"\n",
    "    \"vpc_subnet_self_link\" = \"https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/regions/us-west1/subnetworks/default\"\n",
    "  }\n",
    "  \"opensearch_vm\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"pgbouncer\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"postgres\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"praefect\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"praefect_postgres\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"redis\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"redis_cache\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"redis_persistent\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "  \"sidekiq\" = {\n",
    "    \"data_disk_device_names\" = []\n",
    "    \"external_addresses\" = []\n",
    "    \"internal_addresses\" = {}\n",
    "    \"machine_names\" = []\n",
    "    \"self_links\" = []\n",
    "    \"service_account\" = null\n",
    "    \"zones\" = []\n",
    "  }\n",
    "}\n",
    "\n",
    "real\t0m52.330s\n",
    "user\t0m3.575s\n",
    "sys\t0m0.862s\n",
    "\n",
    "\n",
    "</code></pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## 4. Ansible Playbooks\n",
    "\n",
    "After provisioning the GCP infrastructure, we use Ansible to deploy GitLab into it. The aliases are _similar_ but we run the Ansible playbooks from the main ansible directory, not the environment directory for the individual configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alias cdae='cd ${GET_HOME}/ansible/environments/'\n"
     ]
    }
   ],
   "source": [
    "alias cdae\n",
    "cdae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ansible command to run the playbook with the right inventory is quite long. I have a shell function to do it. It also takes care to start in the correct directory, so we didn't _need_ to use `cdae` first, but it does make it easier to get your bearings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aplayall is a function\n",
      "aplayall () \n",
      "{ \n",
      "    local FUNCDESC='Play all ansible playbooks for specified environment';\n",
      "    [[ -z ${1} ]] && { \n",
      "        error \"${FUNCNAME}: error: must specify an environment from ${GET_HOME}/ansible/environments/\";\n",
      "        return 1\n",
      "    };\n",
      "    E=$(echo \"${1}\"|tr -d '/');\n",
      "    pushd ${GET_HOME}/ansible/;\n",
      "    time ansible-playbook -i environments/${E}/inventory/ playbooks/all.yml;\n",
      "    popd\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "type aplayall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Ansible playall\n",
    "\n",
    "Let's run it to deploy GitLab! This step takes the longest time — about 20 minutes for a single-server \"1k\" reference architecture, and an hour or more for larger environments. Now is a good time to make a coffee or find another task to do.\n",
    "\n",
    "The output will be saved into the notebook, so any problems can be reviewed and then fixed and retried."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "aplayall gl8_1k_v170401"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "\n",
    "/workspaces/get/ansible /workspaces/get/ansible/environments\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "[WARNING]: Found variable using reserved name: tags\n",
    "[WARNING]: Found variable using reserved name: name\n",
    "\n",
    "TASK [common : Gather facts (common)] ******************************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Mark facts gathered] ********************************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Gather package facts (common)] **********************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Check if GitLab repository exists] ******************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Install system packages (Ubuntu / Debian)] **********************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Install Python packages] ****************************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Configure TCP keepalive settings] *******************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Get Group Data Disks Config] ************************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Create select GitLab dirs if missing] ***************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1] => (item={'path': '/etc/gitlab', 'mode': '0775'})\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1] => (item={'path': '/etc/gitlab/ssl', 'mode': '0755'})\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1] => (item={'path': '/etc/gitlab/trusted-certs', 'mode': '0755'})\n",
    "\n",
    "TASK [common : Create skip-unmigrated-data-check file] *************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Get GitLab json config file stats if it exists] *****************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Download GitLab repository installation script] *****************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Install GitLab repository] **************************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Install GitLab repo package (deb)] ******************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Lock GitLab package updates] ************************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Install unattended-upgrades package] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Run Automatic Security Upgrades directly (Ubuntu / Debian)] *****\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Check if custom config exists] **********************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "TASK [common : Remove old Custom Config if not configured] *********************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Check if Custom Tasks file exists] ******************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "TASK [common : Mark common has run] ********************************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [haproxy_*] ***************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [gitlab_nfs] **************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [consul] ******************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [postgres] ****************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [pgbouncer] ***************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [redis,redis_cache_*,redis_persistent_*] **********************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [gitaly] ******************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [praefect_postgres] *******************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [praefect] ****************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "TASK [Load Secrets if existing] ************************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "PLAY [gitlab_rails] ************************************************************\n",
    "\n",
    "TASK [Propagate Secrets if existing] *******************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Check if custom config exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "TASK [gitlab_rails : Setup GitLab deploy node config file with DB Migrations] ***\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Reconfigure GitLab deploy node] ***************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Setup all GitLab Rails config files] **********************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Remove old Custom Config if not configured] ***************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Reconfigure all GitLab Rails] *****************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [Propagate Secrets if new or changed] *************************************\n",
    "included: common for mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "\n",
    "TASK [common : Check if GitLab Secrets file exists] ****************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Lookup GitLab Secrets file from first node] *********************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [common : Save GitLab Secrets file contents] ******************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "TASK [common : Mark that secrets have been collected after first reconfigure] ***\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "TASK [common : Write GitLab Secrets file] **************************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Configure SSH Fast Lookup if OpenSSH service being used] ***\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Reload OpenSSH service if being used] *********************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Restart GitLab] *******************************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Enable Incremental Logging (Object Storage)] **************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Create skip-auto-reconfigure file] ************************\n",
    "changed: [mjl5658-gl8-1k-v170401-gitlab-rails-1]\n",
    "\n",
    "TASK [gitlab_rails : Check if Custom Tasks file exists] ************************\n",
    "ok: [mjl5658-gl8-1k-v170401-gitlab-rails-1 -> localhost]\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "PLAY [sidekiq] *****************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "PLAY [monitor] *****************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "PLAY [localhost] ***************************************************************\n",
    "\n",
    "PLAY [all:!gitlab_*_node_pool:!ungrouped] **************************************\n",
    "\n",
    "PLAY [opensearch] **************************************************************\n",
    "skipping: no hosts matched\n",
    "\n",
    "PLAY [localhost] ***************************************************************\n",
    "FAILED - RETRYING: [localhost]: Wait for GitLab to be available (30 retries left).\n",
    "FAILED - RETRYING: [localhost]: Wait for GitLab to be available (29 retries left).\n",
    "FAILED - RETRYING: [localhost]: Wait for GitLab to be available (28 retries left).\n",
    "FAILED - RETRYING: [localhost]: Wait for GitLab to be available (27 retries left).\n",
    "FAILED - RETRYING: [localhost]: Wait for GitLab to be available (26 retries left).\n",
    "\n",
    "TASK [post_configure : Wait for GitLab to be available] ************************\n",
    "ok: [localhost]\n",
    "\n",
    "TASK [post_configure : Get and save Environment Settings and License plan via GitLab Rails] ***\n",
    "changed: [localhost -> mjl5658-gl8-1k-v170401-gitlab-rails-1(34.168.4.189)]\n",
    "\n",
    "TASK [post_configure : Save Environment Settings] ******************************\n",
    "ok: [localhost]\n",
    "\n",
    "TASK [post_configure : Check for existing license or configure new subscription / license via GitLab Rails] ***\n",
    "changed: [localhost -> mjl5658-gl8-1k-v170401-gitlab-rails-1(34.168.4.189)]\n",
    "\n",
    "TASK [post_configure : Save license plan] **************************************\n",
    "ok: [localhost]\n",
    "\n",
    "TASK [post_configure : Disable Write to \"authorized_keys\" file setting via GitLab Rails] ***\n",
    "changed: [localhost -> mjl5658-gl8-1k-v170401-gitlab-rails-1(34.168.4.189)]\n",
    "\n",
    "TASK [post_configure : Enable 'gitlab_environment_toolkit_instance' setting via GitLab Rails] ***\n",
    "changed: [localhost -> mjl5658-gl8-1k-v170401-gitlab-rails-1(34.168.4.189)]\n",
    "\n",
    "TASK [post_configure : Check if Custom Tasks file exists] **********************\n",
    "ok: [localhost]\n",
    "\n",
    "PLAY RECAP *********************************************************************\n",
    "localhost                  : ok=8    changed=4    unreachable=0    failed=0    skipped=152  rescued=0    ignored=0   \n",
    "mjl5658-gl8-1k-v170401-gitlab-rails-1 : ok=59   changed=18   unreachable=0    failed=0    skipped=232  rescued=0    ignored=0   \n",
    "\n",
    "\n",
    "real\t20m17.253s\n",
    "user\t0m49.661s\n",
    "sys\t0m20.607s\n",
    "/workspaces/get/ansible/environments\n",
    "\n",
    "</code></pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Confirm environment\n",
    "\n",
    "After we have installed GitLab, let's confirm that it is up and okay, before moving on to using it (testing, experimenting, or replicating our customer's problem)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "curl -fssL http://$(gaddress-ip gl8)/-/health\n",
    "curl -fssL http://$(gaddress-ip gl8)/-/readiness?all=1|jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>▹ Toggle output</summary>\n",
    "\n",
    "<pre><code>\n",
    "\n",
    "GitLab OK\n",
    "{\n",
    "  \"status\": \"ok\",\n",
    "  \"master_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"db_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"buffered_counter_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"cache_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"db_load_balancing_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"feature_flag_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"queues_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"queues_metadata_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"rate_limiting_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"repository_cache_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"sessions_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"shared_state_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"trace_chunks_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"chat_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"workhorse_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\"\n",
    "    }\n",
    "  ],\n",
    "  \"gitaly_check\": [\n",
    "    {\n",
    "      \"status\": \"ok\",\n",
    "      \"labels\": {\n",
    "        \"shard\": \"default\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "</code></pre>\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Login details\n",
    "\n",
    "We can logon to the instance at its `external_url`, which will be the IP address. There is a single GitLab `root` administrator account, and it will have the randomly generated password. These values are in the Ansible `vars.yml`. I also have a function which prints those details.\n",
    "\n",
    "---\n",
    "\n",
    "> 📝 ***Note***: this is not a safe way to store the secrets. In future I will be looking into using Ansible Vault to encrypt the secrets. The shell function will have to be updated then as well, of course.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gitlab-url is a function\n",
      "gitlab-url () \n",
      "{ \n",
      "    local FUNCDESC='Print the URL and default root password for a GitLab instance';\n",
      "    if [[ -z ${1} ]]; then\n",
      "        echo \"Usage: ${FUNCDESC} «name_pattern»\" ${FUNCDESC};\n",
      "        return 1;\n",
      "    fi;\n",
      "    E=$(echo \"${1}\"|tr -d '/');\n",
      "    yq '.all.vars.external_url, .all.vars.gitlab_root_password' ${GET_CONT}/ANSIBLE/${E}/inventory/vars.yml\n",
      "}\n",
      "http://34.168.4.189\n",
      "ur5iY5Cea8yez9Oo\n"
     ]
    }
   ],
   "source": [
    "type gitlab-url\n",
    "gitlab-url gl8_1k_v170401"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Clean up environment\n",
    "\n",
    "When you no longer need this GitLab environment, you can turn it off, or destroy it.\n",
    "\n",
    "You can stop and start the environment using `gcloud compute instances stop`|`start INSTANCE_NAME`.\n",
    "\n",
    "The full INSTANCE_NAME is the name as defined in the configuration, which will be the same as the directory name, preceded by your global _ident_ string, and followed by the GET _role_. For my single-server test instance it is `mjl5658-DIRECTORY_NAME-gitlab-rails-1`.\n",
    "\n",
    "```\n",
    "gcloud compute instances stop mjl5658-gl8-1k-v170410-gitlab-rails-1\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "> 📝 ***Note*** for larger environments, there will be multiple GCP compute instances, and they must each be started or stopped. I have [a ruby script](https://gitlab.com/mlockhart/lab/-/blob/master/googleplex/glplex) that uses `gcloud` with pattern matching, so that I can just use the prefix to find all of the instances (e.g. all matching `gl8`). Usually I run this script directly from my MacBook, not within the container. Unfortunately, I still have [an outstanding issue](https://gitlab.com/mlockhart/get-container/-/issues/2) to migrate that into the GET container. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.0 Fetch instances\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Since my convenience script isn't available to this notebook in the container, here is an approximate equivalent in bash:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "PATTERN=gl8\n",
    "STATUS=RUNNING # or TERMINATED\n",
    "INSTANCES=$(gcloud compute instances list \\\n",
    "  --filter=\"(name~$PATTERN AND status=$STATUS)\" \\\n",
    "  --zones=\"$GCP_ZONE\" \\\n",
    "  --format=\"value[terminator=\\\" \\\"](name)\")\n",
    "echo $INSTANCES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "mjl5658-gl8-1k-v170401-gitlab-rails-1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Stop instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Use `gcloud` to stop each instance. It can take multiple arguments, so you don't need to loop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stopping instance(s) mjl5658-gl8-1k-v170401-gitlab-rails-1...done.             \n",
      "Updated [https://compute.googleapis.com/compute/v1/projects/mlockhart-56581c10/zones/us-west1-c/instances/mjl5658-gl8-1k-v170401-gitlab-rails-1].\n"
     ]
    }
   ],
   "source": [
    "gcloud compute instances stop $INSTANCES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Destroy environment\n",
    "\n",
    "(It's not necessary to stop it first, but I've included both steps so you can (re) run either one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `terraform destroy` command is interactive, requiring manual confirmation, so I can't run it from the notebook.\n",
    "\n",
    "You would do the following:\n",
    "\n",
    "1. `cdte`\n",
    "1. `cd gl6_1k_v170401  #or whichever environment`\n",
    "1. `tf destroy`\n",
    "1. answer `yes`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Release the external IP address\n",
    "\n",
    "You can also release the IP address at this point.  This is also has an interactive confirmation prompt, so here is a transcript:\n",
    "\n",
    "```\n",
    "gaddress delete gl8\n",
    "```\n",
    "```\n",
    "The following addresses will be deleted:\n",
    " - [gl8] in [us-west1]\n",
    "\n",
    "Do you want to continue (Y/n)?  \n",
    "\n",
    "Deleted [https://www.googleapis.com/compute/v1/projects/mlockhart-56581c10/regions/us-west1/addresses/gl8].\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "That's the end of this short demo. I hope you found this instructive for how to employ GET from a Dev Container, to install GitLab into GCP. We did the following:\n",
    "\n",
    "1. Reserve a GCP public IP address\n",
    "1. Create a GET configuration based on the 1k template\n",
    "1. Provision the GCP infrastructure using Terraform\n",
    "1. Install GitLab using Ansible\n",
    "1. Test that GitLab is running\n",
    "1. Clean up\n",
    "    1. Stop the environment's GCP compute instances\n",
    "    1. Destroy the environment using Terraform\n",
    "    1. Release the GCP reserved IP address\n",
    "\n",
    "Other, larger reference architectures are available too. For testing, I often use the 2k ref arch to have separate Sidekiq and Rails servers, or the 3k ref arch for Gitaly Cluster. It's important to remember to use `--economic`|`-E` option for the configure script to contain cost, as these are large environments!\n",
    "\n",
    "You can use [this blog's source code](/blog/2025/get-container-demo.ipynb) as a starting point. Just open it in the get-container VS Code dev container. It has been modified somewhat for presentation, so search for the cells that contain `Toggle output`, and remove them. Alternatively, create a new notebook in your get-container and follow along.\n",
    "\n",
    "Happy Hacking."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}