At my last job I was asked to consolidate and automate the de-provisioning of enterprise infrastructure, primarily Virtual Machines. I believe that web apps are the way of the future, so I decided to achieve this feat with a “simple” Django application, but considering the requirements (listed below), it would be anything but simple.

Requirements

  • Authenticate users against Active Directory
  • Live search VMware vSphere for Virtual Machines
  • Automatically remove user-specified VMs
  • Clean up records of deleted VMs in other systems
  • Allow additional tasks to be added at a later date
  • Allow users to search for in-progress and completed decommissions
  • Have a full-featured API
  • Be fast

Technologies Used

  • Python 3.6
  • Django
  • Django Rest Framework
  • Django Channels
  • JavaScript
  • jQuery
  • Docker
  • Redis

Here’s a quick run down of functionality:

The Login Page displaying an “Invalid Permissions” error

The user is authenticated against AD via a custom Django Authentication Backend.  The backend also checks that the authenticated user has permission to modify certain objects in vSphere.

Decommission Request with three VMs

The primary purpose of the application is to allow users to build “Decommission Requests”.  A Decommission Request is composed of one or more VMs and associated options.  Users can search vCenter for VMs to add to the Request.  For safety checks, the application will display additional information about the VM including the machine’s IP Address and Operating System.

After submitting the request, the application will queue the Decommission Request and redirect the user to the “Your Requests” page.

Live display of tasks applied to a VM

Decommissions are dequeued and performed by Task Workers which are additional python processes that run in a different container.  By default, five workers are spawned per Worker container, but this value can be modified with environment variables and the Worker container can be scaled up seamlessly.

A single worker can decommission one VM at a time, so if there are few enough VMs in a Decommission Request, all VMs can be decommissioned concurrently.  Decommissions are built and executed as a series of steps that is dependent on what OS the VM is running.

As a Worker process is executing a decommission, the status of that decommission is displayed, live, on the UI via WebSockets.  If there are warnings or errors during a step, these are also displayed.

At any time, a user can immediately cancel a decommission, but steps that have already been performed are not rolled back.

The Application’s Full-featured API displaying Decommissioned Machines