Today, I finally decided to debug an interesting issue that I have run across while testing the NetApp Docker Volume Plug-in.  Part of the testing that I am doing involves deploying a Docker host, installing the plug-in, and provisioning containers and volumes on NetApp ONTAP Cloud.  All has gone great and very simple to manage.  I created a configuration file and quickly started spinning up new docker volumes.  I did notice something a bit odd though.  Every time that I rebooted my host, I couldn’t get the containers to start.  I had a workaround of manually mounting a NFS export on the host and magically, Docker would start the container.  This was a bit of an annoyance more than anything but as I am about to release an update to the ONTAP Cloud CHEF cookbook and will support the NetApp Docker Volume Plug-in, I think it is time to do some debugging.

In my case, I have a single MySQL container with a single docker volume leveraging the NetApp Docker Volume Plug-in (nDVP).  It is a simple enough configuration and I can leverage my NFS connections back to the ONTAP Cloud system.  In order to support this configuration, I installed the NFS client for my host and the nDVP package.  Within minutes, I had a working solution and rapidly had a ton of NetApp Volumes and Docker Containers spinning along.  The only issue came when I rebooted and the container wouldn’t start.  In fact, I received an error like below (super helpful, right?)

$ docker start mysql
Error response from daemon: error while mounting volume '/var/lib/docker/plugins/231ea1226c6d9f7285201816e5716ca3c2f59376a009ab0e3957b369d87b3b05/rootfs': 
VolumeDriver.Mount: Problem attaching docker volume: netappdvp_mysql_data mountpoint: /var/lib/docker-volumes/netapp/netappdvp_mysql_data error: Problem 
mounting volume: netappdvp_mysql_data mountpoint: /var/lib/docker-volumes/netapp/netappdvp_mysql_data error: exit status 32
Error: failed to start containers: mysql

After picking my head off the desk, I started looking into the error.  Like a scene from the Usual Suspects, I dove into the syslog and docker logs.  Nothing but a very unusual error and what was exit status 32?  I did do some research into how to debug the nDVP package.  This information is available in the official nDVP Documentation and pretty handy for looking into problems.  In my case, there was nothing.

# Find the netappdvp container ID
$ docker-runc list

# View the Plug-in Logs
$ docker-runc exec -t <container id> cat /var/log/netappdvp/netapp.log

In desperation, I ran a packet trace using tcpdump and I noticed that there was no NFS traffic.  At this point, I concluded that there is a configuration issue in the host.  I had pretty much confirmed that when I came up with my earlier workaround.  After a quick trip to the helpful folks at the official NetApp DevOps Slack channel – the Pub, we determined that there was something about Ubuntu (my host) and the Docker service.  As it turns out, Docker has no dependency on RPCBind or the NFS utilities and since nDVP is a plug-in, there is no way for NetApp to add the solution in their code.  We found that I need to add a systemd override file for the docker.service and set the dependency on rpcbind.service.

Here is an example of what I did and how I solved my issue:

$ vi /etc/systemd/system/docker.service.d/netappdvp.conf
[Unit]
Requires=rpcbind.service

$ systemctl daemon-reload # Reload the system configurations
$ systemctl list-dependencies docker | grep -i rpc
● ├─rpcbind.service

$ systemctl restart docker # Restart will apply the configuration