<!DOCTYPE HTML>
<html lang="en-US">
<head>
<title>WARNING - WIP - NOT FOR GENERAL USE - README for Quickstart Job Zuul-based Reproducer Script</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
.collapsible {
  background-color: #777;
  color: white;
  cursor: pointer;
  padding: 18px;
  width: 100%;
  border: none;
  text-align: left;
  outline: none;
  font-size: 15px;
}

.active, .collapsible:hover {
  background-color: #555;
}

.content {
  padding: 0 18px;
  max-height: 0;
  overflow: hidden;
  transition: max-height 0.2s ease-out;
  background-color: #f1f1f1;
}
</style>
</head>
</head>
<body>
<h1> WARNING - WIP - NOT FOR GENERAL USE - How to reproduce a job result using the Zuul-based reproducer playbooks and script</h1>
<p>Check the top-level logs directory for a tar file <b>reproducer-zuul-based-quickstart.tar</b>.
This tar archive contains three files:
<ul>
<li><code>launcher-env-setup-playbook.yaml</code> - a playbook to clone repos and set up
an Ansible environment to launch jobs</li>
<li><code>launcher-playbook.yaml - a playbook that installs and starts up Zuul, Gerrit and
other required services, and also creates and launches a reproducer job using
the same dlrn hashes as in CI</code></li>
<li><code>reproducer-zuul-based-quickstart.sh - a Bash script to provide an interface
to run the above playbooks with various options</code></li>
</ul>
Running these playbooks will set up Zuul, Gerrit, and related services in containers
on the local system, and will then create a reproducer job which can be run on a
cloud personal tenant or via libvirt on the local system.
If no <b>reproducer-zuul-based-quickstart.tar</b> file is found, that usually means an
infra failure before Quickstart could start or a problem in collecting logs.
Check on IRC Freenode channel <i>#tripleo</i> to see if there's an ongoing infra
issue.</p>

<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
<div class="content">
<p>
Prior to running the Zuul-based reproducer, the following setup steps are needed -
( only need to be done once per local system):

<ul>
<li>Create a clouds.yaml - for example:
<pre>
<code>mkdir -p ~/.config/openstack
vi ~/.config/openstack/clouds.yaml
clouds:
  rdo-cloud:
    identity_api_version: 3
    region_name: regionOne
    auth:
      auth_url: https://phx2.cloud.rdoproject.org:13000/v3
      password: xxxxxxx
      project_name: user_name
      username:  user_name
      user_domain_name: Default
      project_domain_name: Default
    regions:
      - name: regionOne
        values:
          networks:
            - name: 38.145.32.0/22
              routes_externally: true
            - name: private

</code>
</pre>
</ul>
<ul>
<li> Test your cloud.yaml by running:
<pre>
<code>
 openstack --os-cloud rdo-cloud image list
</code>
</pre>
</ul>
<ul>
<li>Note that you will need a network named 'private' in the tenant connected to an
router that is connected to the external network.</li>
<li>Create an ssh key pair without a pass phrase:</li>
  <ul>
  <li>The ssh key pair is configurable - see variables <code>ssh_path</code></li>
  <li><code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
  <li> <b>Note:</b> Reference FAQ below when creating ssh keys, some keys fail when used w/ gerrit
  </ul>
</p>
</div>

<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
<div class="content">
<p>
<ul>
<li> wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file. </li>
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will set up
clone repos and set up 'roles', 'playbooks', and 'library' directories
from which the tasks will be executed.</li>
<li> Once the files are extracted from the archive, you can run the Bash script
with the various options. Run the script with <code>--help</code> option to
output a list of available run options. </li>
</ul>
<b>Note:</b> the local $USERNAME idealy matches the openstack and rdo gerrit user id's
If not pay extra attention to the -ug and -rg options.
<ul>
<code>
<pre>
Options:
  -w, --workspace dir
                      directory where the repos and directories
                      are created. Defaults to creating a directory in /tmp
  -l, --libvirt
                      Runs a 2-node multinode job or singlenode job on a
                      single virthost using libvirt to create the nodes.
                      If a singlenode reproducer is run, two VMs will still be created.
  -c, --cloud-name
                      Host cloud, specified in the clouds.yaml, to target
                      Defaults to rdo-cloud
  -cp, --clouds-yaml-path
                      Full path to the clouds.yaml file
                      Defaults to /home/$USERNAME/.config/openstack/clouds.yaml
  -ok, --ovb-key-name
                      Name of the key to use in the host tenant for OVB deployments
                      Defaults to tripleo-ci-team
  -f, --force-post-failure
                      Force job to fail so that the nodes will be held.
                      Temporary solution
  -ug, --upstream-gerrit-user
                      Set the upstream gerrit user required to clone repos.
                      Defaults to the current $USERNAME
  -rg, --rdo-gerrit-user
                      Set the upstream RDO user required to clone RDO-related repos.
                      Defaults to the current $USERNAME
  -k, --ssh-key
                      private ssh key used to set up an access nodes.
                      Defaults to id_rsa.
  -kp, --ssh-key-public
                      public ssh key used to set up an access nodes.
                      Defaults to id_rsa.pub
  -skp, --ssh-key-path
                      Path to directory where user ssh keys are stored.
                      Defaults to /home/$USERNAME/.ssh
  -ugk, --upstream-gerrit-key
                      Set the upstream gerrit private key.
                      Defaults to the id_rsa.
  -rgk, --rdo-gerrit-key
                      Set the upstream RDO user key.
                      Defaults to the id_rsa.
  -e, --extra-params
                      File or/and parameters used to override default
                      parameters for playbooks. Multiple files
                      can be passed [-e @file1.yml -e @file2.yml ...]
                      and arguments [-e var=value -e var2=value2 ...]
  -h, --help          print this help and exit

</code>
</pre>
</ul>
</p>
</div>

<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
<div class="content">
<p>
The Bash script is not required although it is the supported path.
The playbooks can be run directly using <code>ansible-playbook</code>.
The Bash script contains some package installation and Docker user/group setup
that you will need to cover prior to running the playbooks directly.
Please see the Bash scripts for details, especially if you are running the
reproducer for the first time on teh local environment.
The playbooks can be run as follows:
<ul>
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
</ul>
The <code>launcher-playbook.yaml</code> includes comments on how to run the playbook
with the same options listed in the Bash script section above.
</p>
</div>

<button class="collapsible">Running the Zuul-based reproducer - autohold nodes for debugging</button>
<div class="content">
<p>
The reproducer can be launched such that if there is an error the environment will
remain up and running available to the developer to debug.

<ul>
<li>For example:</li>

</ul>
<li><code> cd ~/tripleo-ci-reproducer </code></li>
<li><code>docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job tripleo-ci-centos-7-standalone --reason debug --project test1</code></li>
</p>
</div>

<button class="collapsible">Ansible tags and running repeatedly</button>
<div class="content">
<p>

<ul>
<li> The Bash scripts runs with <code>--tags all</code> passed to the
<code>launcher-playbook.yaml</code>.
</li>
<li>
After the initial docker setup has run, it is not necessary to run through
those setup steps for every reproducer. The <code>launcher-playbook.yaml</code>
can be run with <code>--skip-tags start</code> after the initial run.
</li>
</p>
</div>

<button class="collapsible">Monitoring the launched job</button>
<div class="content">
<p>
The <code>launcher-playbook.yaml</code> will create and push a job
to a test1 testproject repo in a temp directory. This job will start and the
related Zuul infrastructure logs can be viewed in:</br>
<pre>
<code>cd ~/tripleo-ci-reproducer
docker-compose logs --tail=10 -f
# Just view the scheduler and executor logs:
docker-compose logs --tail=10 -f scheduler executor</code>
</pre>
</br>
To see the job progress, you can access the web consoles of:
<ul>
<li>Gerrit: http://localhost:8080</li>
<li>Zuul: http://localhost:9000</li>
<li>Logs: http://localhost:8000</li>
</ul>
If you are accessing a remote machine, replace <code>localhost</code> with the ip address.
</p>
</div>

<button class="collapsible">Known Issues and FAQ</button>
<div class="content">
<p>
<ul>
<li>Gerrit keys<br/>
If your ssh keys have being created with a new version of OpenSSH, Zuul will
fail when running with these keys as there is a Paramiko bug [1].
It is recommended  that keys be generated as follows:
<code>ssh-keygen -m PEM -t rsa</code> - accepting the passwordless option.
Then update the public keys used with Gerrit upstream and RDO Gerrit.</br>
[1] https://storyboard.openstack.org/#!/story/2004842</li>
</ul>
<ul>
<li>If you currently do not have access to upstream-cloudinit-[centos-7, fedora-28] : </li>
<code>
<pre>
source the openrc from the openstack-nodepool tenant
openstack image list | grep upstream-cloudinit
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
</code>
</pre>
</ul>
</p>
</p>
</div>

<script>
  var coll = document.getElementsByClassName("collapsible");
  var i;

  for (i = 0; i < coll.length; i++) {
    coll[i].addEventListener("click", function() {
      this.classList.toggle("active");
      var content = this.nextElementSibling;
      if (content.style.maxHeight){
        content.style.maxHeight = null;
      } else {
        content.style.maxHeight = content.scrollHeight + "px";
      }
    });
  }
  </script>

</body>
</html>
