Troubleshooting OpenStack

OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter. However, OpenStack like all software will have bugs that we are not able to solve ourselves. In that case, we will show you how to gather the required information so that the OpenStack community can identify bugs and suggest fixes that are important in ensuring those bugs or issues are dealt with quickly and efficiently.

Understanding logging in Troubleshooting Openstack

• Logging is important in all computer systems, but the more complex the system, the more you rely on logging to be able to spot problems and cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and you are able to submit relevant log entries back to the community to help fix bugs.
• Log in as the root user onto the appropriate servers where the OpenStack services are installed. This makes troubleshooting easier as root privileges are required to view all the logs.
• OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs:

#1.OpenStack Compute Services Logs

Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user (or use sudo privileges when accessing the files). The following is a list of services and their corresponding logs. Note that not all logs exist on all servers. For example, compute.log exists on your compute hosts only:

nova-compute: /var/log/nova/nova-compute.log

Log entries regarding the spinning up and running of the instances

nova-network: /var/log/nova/nova-network.log

Log entries regarding network state, assignment, routing, and security groups

nova-manage: /var/log/nova/nova-manage.log

Log entries produced when running the nova-manage command

nova-conductor: /var/log/nova/nova-conductor.log

Log entries regarding services making requests for database information

nova-scheduler: /var/log/nova/nova-scheduler.log

Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue

nova-api: /var/log/nova/nova-api.log

Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack

nova-cert: /var/log/nova/nova-cert.log

Entries regarding the nova-cert process

nova-console: /var/log/nova/nova-console.log

Details about the nova-console VNC service

nova-consoleauth: /var/log/nova/nova-consoleauth.log

Authentication details related to the nova-console service

nova-dhcpbridge: /var/log/nova/nova-dhcpbridge.log

Network information regarding the dhcpbridge service

#2.OpenStack Dashboard logs

OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service.

#3.OpenStack Storage logs

OpenStack Object Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages.
The OpenStack Block Storage service, Cinder, will produce logs in /var/log/cinder by default. The following list is a breakdown of the log files:

cinder-api: /var/log/cinder/cinder-api.log

Details about the cinder-api service

cinder-scheduler: /var/log/cinder-scheduler.log

Details related to the operation of the Cinder scheduling service

cinder-volume: /var/log/cinder/cinder-volume.log

Log entries related to the Cinder volume service

[Related Article: OpenStack Networking]

#4.OpenStack Identity logs

The OpenStack Identity service, Keystone, writes its logging information to /var/log/keystone/keystone.log. Depending on how you have Keystone configured, the information in this log file can be very sparse to extremely verbose including complete plaintext requests.
OpenStack Image Service logs
The OpenStack Image Service Glance stores its logs in /var/log/glance/*.log with a separate log file for each service. The following is a list of the default log files:

api: /var/log/glance/api.log

Entries related to the glance API

registry: /var/log/glance/registry.log

Log entries related to the Glance registry service. Things like metadata updates and access will be stored here, depending on your logging configuration.

#5.OpenStack Network Service logs

OpenStack Networking Service, formerly Quantum, now Neutron, stores its log files in /var/log/quantum/*.log with a separate log file for each service. The following is a list of the corresponding logs:

dhcp-agent: /var/log/quantum/dhcp-agent.log

Log entries pertaining to the dhcp-agent

l3-agent: /var/log/quantum/l3-agent.log

Log entries related to the l3 agent and its functionality

metadata-agent: /var/log/quantum/metadata-agent.log

This file contains log entries related to requests Quantum has proxied to the Nova metadata service.

openvswitch-agent: /var/log/quantum/openvswitch-agent.log

Entries related the

[DEFAULT]
#show more verbose log output (sets INFO log level output)
verbose=FALSE

#Show debugging output in logs(sets DEBUG log level output)
debug=FALSE

operation of Open vSwitch. When implementing OpenStack Networking, if you use a different plugin, its log file will be named accordingly.

server: /var/log/quantum/server.log

Details and entries related to the quantum API service

OpenVSwitch Server: /var/log/openvswitch/ovs-vswitchd.log

Details and entries related to the OpenVSwitch Switch Daemon

Changing log levels

By default, each OpenStack service has a sane level of logging, which is determined by the level set as a Warning. That is, it will log enough information to provide you the status of the running system as well as some basic troubleshooting information. However, there will be times that you need to adjust the logging verbosity either up or down to help diagnose an issue or reduce logging noise.
As each service can be configured similarly, we will show you how to make these changes on the OpenStack Compute service.

Log-level settings in OpenStack Compute services

To do this, log into the box where the OpenStack Compute service is running and execute the following commands:

sudo vim /etc/nova/logging.conf

[Related Article: Troubleshooting OpenStack Compute services]

Change the following log levels to either DEBUG, INFO, or WARNING in any of the services listed:

[logger_root]
Level=WARNING
handlers=null

[logger_nova]
Level=INFO
handlers=stderr
qualname=nova

Other services such as Glance and Keystone currently have their log-level settings within their main configuration files such as /etc/glance/glance-api.conf. Adjust the log levels by altering the following lines to achieve INFO or DEBUG levels:

[DEFAULT]
#show more verbose log outputs(sets INFO Log level output)
verbose=False

#Show debugging output in logs (sets DEBUG Log level output)
debug=False

Restart the relevant service to pick up the log-level change.

Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log and managing those logs to allow someone to identify problems quickly and easily are important