GuruMeditationReport
目录
Guru Meditation Reports
当 OpenStack 的（生产）部署出现问题时，收集调试数据是解决问题和最终解决问题的关键第一步。Nova 已经广泛使用了日志记录功能，产生大量的数据。然而，这并不能使管理员获得系统当前实时状态的准确视图。例如，正在运行哪些线程，哪些配置参数生效，等等。eventlet 后门设施为任何基于 eventlet 的进程提供了一个交互式 shell 界面，允许管理员 telnet 到预定义的端口并执行各种命令。这可用于收集必要的状态信息，但它有许多限制。
- 每个在主机上运行的服务都需要在不同的 TCP 端口上运行后门，并且管理员必须记住哪个进程在监听哪个端口。搞错的话会发生非常糟糕的事情。
- 后门需要在进程启动时启用。如果问题出现之前没有这样做，管理员就无能为力，因为重启服务以启用后门会丢失所需的关键状态。
- 后门 shell 功能过于强大。通过呈现一个交互式 python shell，给管理员带来了过多的负担，需要找到正确的数据，而不会造成问题。
- 后门 shell 是一个有效的“root shell”，可以通过不安全的 TCP 端口提供，因此任何理智的管理员都不会在生产系统上启用它。
Error Report Framework
为了解决上述问题，本页概述了一个通用错误报告生成框架的设计，称为“guru meditation report”（参见 http://en.wikipedia.org/wiki/Guru_Meditation）。
- Models: 这些类定义了各种有趣状态的结构化数据。例如，堆栈跟踪、线程、配置参数、软件包版本信息等。它们能够序列化为 XML / JSON 或纯文本表示形式
- Generators: 这些类用于使用系统的当前运行时状态填充模型类
将为所有 OpenStack 服务提供许多标准模型/生成器
- StackTraceModel: 任何包含堆栈跟踪的模型的基类
- ThreadModel: 关于线程的信息类
- ExceptionModel: 关于捕获的异常的信息类
- ConfigModel: 关于配置文件设置的信息类
- PackageModel: 关于 vendor/product/version/package 信息类
每个 OpenStack 项目将能够注册进一步的生成器类，以提供自定义的项目特定数据。
Integration with apps
每个长期运行的服务进程都应该调用一个安装信号处理程序的函数，该函数将在收到 SIGUSR1 时触发 guru meditation 框架。这将导致该进程将其当前状态的完整报告转储到 stderr。
对于处理 RPC 进程的进程，也可能需要在 RPC 请求调度器中安装某种钩子，以便在处理请求导致未捕获的异常时保存 guru meditation 报告。可以将这些报告保存到周知目录（/var/log/openstack/<project>/<service>/）中，供系统管理员或自动化错误分析工具稍后分析。
Example Report
========================================================================
==== Guru meditation report ====
========================================================================
UUID: 6d4faf99-480c-4d0c-90be-b7d090ecd83e
Time: Mon, 18 Feb 2013 18:01:11 +0000
========================================================================
==== Config ====
========================================================================
DEFAULT:
allow_resize_to_same_host=True
allow_same_net_traffic=True
allowed_direct_url_schemes=[]
allowed_rpc_exception_modules=['nova.openstack.common.exception', 'nova.exception', 'cinder.exception', 'exceptions']
api_paste_config=/etc/nova/api-paste.ini
api_rate_limit=True
auth_strategy=keystone
auto_assign_floating_ip=False
backdoor_port=None
bandwidth_poll_interval=600
base_dir_name=_base
bindir=/home/berrange/src/cloud/nova/bin
block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC
boot_script_template=/home/berrange/src/cloud/nova/nova/cloudpipe/bootscript.template
ca_file=cacert.pem
ca_path=/home/berrange/src/cloud/data/nova/CA
cert_manager=nova.cert.manager.CertManager
checksum_base_images=False
checksum_interval_seconds=3600
cinder_api_insecure=False
cinder_catalog_info=volume:cinder:publicURL
cinder_cross_az_attach=True
cinder_endpoint_template=None
cinder_http_retries=3
cnt_vpn_clients=0
compute_api_class=nova.compute.api.API
compute_driver=libvirt.LibvirtDriver
compute_manager=nova.compute.manager.ComputeManager
compute_stats_class=nova.compute.stats.Stats
compute_topic=compute
config_dir=None
config_drive_format=iso9660
config_drive_skip_versions=1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01
config_drive_tempdir=None
config_file=['/etc/nova/nova.conf']
console_host=mustard.gsslab.fab.redhat.com
console_manager=nova.console.manager.ConsoleProxyManager
console_topic=console
consoleauth_topic=consoleauth
control_exchange=nova
create_unique_mac_address_attempts=5
crl_file=crl.pem
db_backend=sqlalchemy
db_driver=nova.db
debug=True
default_access_ip_network_name=None
default_availability_zone=nova
default_ephemeral_format=None
default_floating_pool=nova
default_instance_type=m1.small
default_log_levels=['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.wsgi.server=WARN']
default_notification_level=INFO
default_publisher_id=mustard.gsslab.fab.redhat.com
default_schedule_zone=None
defer_iptables_apply=False
dhcp_domain=novalocal
dhcp_lease_time=120
dhcpbridge=/home/berrange/src/cloud/nova/bin/nova-dhcpbridge
dhcpbridge_flagfile=/etc/nova/nova.conf
disable_process_locking=False
dmz_cidr=[]
dmz_mask=255.255.255.0
dmz_net=10.0.0.0
dns_server=[]
dns_update_periodic_interval=-1
dnsmasq_config_file=
ec2_listen=0.0.0.0
ec2_listen_port=8773
ec2_private_dns_show_ip=False
ec2_strict_validation=True
ec2_timestamp_expiry=300
ec2_workers=None
enable_new_services=True
enabled_apis=['ec2', 'osapi_compute', 'metadata']
enabled_ssl_apis=[]
fake_call=False
fake_network=False
fake_rabbit=False
fatal_deprecations=False
fatal_exception_format_errors=False
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
fixed_ip_disassociate_timeout=600
fixed_range=10.0.0.0/24
fixed_range_v6=fd00::/48
flat_injected=False
flat_interface=eth0
flat_network_bridge=br100
flat_network_dns=8.8.4.4
floating_ip_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
force_config_drive=None
force_dhcp_release=True
force_raw_images=True
forward_bridge_interface=['all']
gateway=None
gateway_v6=None
glance_api_insecure=False
glance_api_servers=['10.33.8.112:9292']
glance_host=10.33.8.112
glance_num_retries=0
glance_port=9292
glance_protocol=http
glusterfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
heal_instance_info_cache_interval=60
host=mustard.gsslab.fab.redhat.com
host_state_interval=120
image_cache_manager_interval=2400
image_info_filename_pattern=/home/berrange/src/cloud/data/nova/instances/_base/%(image)s.info
injected_network_template=/home/berrange/src/cloud/nova/nova/virt/interfaces.template
instance_build_timeout=0
instance_dns_domain=
instance_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
instance_format=[instance: %(uuid)s]
instance_name_template=instance-%08x
instance_usage_audit=False
instance_usage_audit_period=month
instance_uuid_format=[instance: %(uuid)s]
instances_path=/home/berrange/src/cloud/data/nova/instances
internal_service_availability_zone=internal
iptables_bottom_regex=
iptables_top_regex=
ipv6_backend=rfc2462
key_file=private/cakey.pem
keys_path=/home/berrange/src/cloud/data/nova/keys
keystone_ec2_url=https://:5000/v2.0/ec2tokens
l3_lib=nova.network.l3.LinuxNetL3
libvirt_cpu_mode=none
libvirt_cpu_model=None
libvirt_disk_prefix=None
libvirt_images_type=default
libvirt_images_volume_group=None
libvirt_inject_key=True
libvirt_inject_partition=1
libvirt_inject_password=False
libvirt_lvm_snapshot_size=1000
libvirt_nonblocking=True
libvirt_ovs_bridge=br-int
libvirt_snapshot_compression=False
libvirt_snapshots_directory=/home/berrange/src/cloud/data/nova/instances/snapshots
libvirt_sparse_logical_volumes=False
libvirt_type=kvm
libvirt_uri=
libvirt_use_virtio_for_bridges=True
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
libvirt_volume_drivers=['iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver', 'local=nova.virt.libvirt.volume.LibvirtVolumeDriver', 'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver', 'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver', 'aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver', 'glusterfs=nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver']
libvirt_wait_soft_reboot_seconds=120
linuxnet_interface_driver=nova.network.linux_net.LinuxBridgeInterfaceDriver
linuxnet_ovs_integration_bridge=br-int
live_migration_bandwidth=0
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER
live_migration_retry_count=30
live_migration_uri=qemu+tcp://%s/system
lock_path=/home/berrange/src/cloud/data/nova
lockout_attempts=5
lockout_minutes=15
lockout_window=15
log_config=None
log_date_format=%Y-%m-%d %H:%M:%S
log_dir=None
log_file=None
log_format=%(asctime)s %(levelname)8s [%(name)s] %(message)s
logfile_mode=0644
logging_context_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix=from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d
logging_default_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_exception_prefix=%(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
max_age=0
memcached_servers=None
metadata_host=10.33.8.112
metadata_listen=0.0.0.0
metadata_listen_port=8775
metadata_manager=nova.api.manager.MetadataManager
metadata_port=8775
metadata_workers=None
mkisofs_cmd=genisoimage
monkey_patch=False
monkey_patch_modules=['nova.api.ec2.cloud:nova.openstack.common.notifier.api.notify_decorator', 'nova.compute.api:nova.openstack.common.notifier.api.notify_decorator']
multi_host=False
my_ip=10.33.8.112
network_api_class=nova.network.api.API
network_device_mtu=None
network_driver=nova.network.linux_net
network_manager=nova.network.manager.FlatDHCPManager
network_size=256
network_topic=network
networks_path=/home/berrange/src/cloud/data/nova/networks
nfs_mount_options=None
nfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
non_inheritable_image_properties=['cache_in_nova', 'bittorrent']
notification_driver=[]
notify_api_faults=False
notify_on_any_change=False
notify_on_state_change=None
novncproxy_base_url=http://10.33.8.112:6080/vnc_auto.html
null_kernel=nokernel
num_aoe_discover_tries=3
num_iscsi_scan_tries=3
num_networks=1
os_region_name=None
osapi_compute_listen=0.0.0.0
osapi_compute_listen_port=8774
osapi_compute_workers=None
password_length=12
periodic_enable=True
periodic_fuzzy_delay=60
policy_default_rule=default
policy_file=policy.json
project_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=project-ca-%.16s-%s
public_interface=br100
publish_errors=False
pybasedir=/home/berrange/src/cloud/nova
qpid_heartbeat=60
qpid_hostname=localhost
qpid_hosts=['localhost:5672']
qpid_password=
qpid_port=5672
qpid_protocol=tcp
qpid_sasl_mechanisms=
qpid_tcp_nodelay=True
qpid_username=
quota_cores=20
quota_driver=nova.quota.DbQuotaDriver
quota_floating_ips=10
quota_injected_file_content_bytes=10240
quota_injected_file_path_bytes=255
quota_injected_files=5
quota_instances=10
quota_key_pairs=100
quota_metadata_items=128
quota_ram=51200
quota_security_group_rules=20
quota_security_groups=10
rbd_secret_uuid=None
rbd_user=None
reboot_timeout=0
reclaim_instance_interval=0
remove_unused_base_images=True
remove_unused_original_minimum_age_seconds=86400
remove_unused_resized_minimum_age_seconds=3600
report_interval=10
rescue_image_id=None
rescue_kernel_id=None
rescue_ramdisk_id=None
rescue_timeout=0
reservation_expire=86400
reserved_host_disk_mb=0
reserved_host_memory_mb=512
resize_confirm_window=0
resume_guests_state_on_host_boot=False
rootwrap_config=/etc/nova/rootwrap.conf
routing_source_ip=10.33.8.112
rpc_backend=nova.openstack.common.rpc.impl_qpid
rpc_cast_timeout=30
rpc_conn_pool_size=30
rpc_response_timeout=60
rpc_thread_pool_size=64
run_external_periodic_tasks=True
running_deleted_instance_action=log
running_deleted_instance_poll_interval=1800
running_deleted_instance_timeout=0
scheduler_manager=nova.scheduler.manager.SchedulerManager
scheduler_topic=scheduler
security_group_api=nova.compute.api.SecurityGroupAPI
security_group_handler=nova.network.sg.NullSecurityGroupHandler
send_arp_for_ha=False
send_arp_for_ha_count=3
service_down_time=60
servicegroup_driver=db
share_dhcp_address=False
snapshot_image_format=None
snapshot_name_template=snapshot-%s
sql_connection=mysql://root:123456@localhost/nova?charset=utf8
sql_connection_debug=0
sql_connection_trace=False
sql_dbpool_enable=False
sql_idle_timeout=3600
sql_max_overflow=None
sql_max_pool_size=5
sql_max_retries=10
sql_min_pool_size=1
sql_retry_interval=10
sqlite_db=nova.sqlite
sqlite_synchronous=True
ssl_ca_file=None
ssl_cert_file=None
ssl_key_file=None
state_path=/home/berrange/src/cloud/data/nova
syslog_log_facility=LOG_USER
tcp_keepidle=600
teardown_unused_network_gateway=False
tempdir=None
until_refresh=0
update_dns_entries=False
use_cow_images=True
use_forwarded_for=False
use_ipv6=False
use_network_dns_servers=False
use_project_ca=False
use_single_default_gateway=False
use_stderr=True
use_syslog=False
use_usb_tablet=True
user_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=%.16s-%.16s-%s
verbose=True
virt_mkfs=['default=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'linux=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'windows=mkfs.ntfs --force --fast --label %(fs_label)s %(target)s']
vlan_interface=eth0
vlan_start=100
vnc_enabled=True
vnc_keymap=en-us
vncserver_listen=127.0.0.1
vncserver_proxyclient_address=127.0.0.1
volume_api_class=nova.volume.cinder.API
volume_usage_poll_interval=0
vpn_image_id=0
vpn_instance_type=m1.tiny
vpn_ip=10.33.8.112
vpn_key_suffix=-vpn
vpn_start=1000
wsgi_log_format=%(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f
xen_hvmloader_path=/usr/lib/xen/boot/hvmloader
xvpvncproxy_base_url=http://10.33.8.112:6081/console
cells:
call_timeout=60
capabilities=['hypervisor=xenserver;kvm', 'os=linux;windows']
enable=False
manager=nova.cells.manager.CellsManager
name=nova
topic=cells
conductor:
manager=nova.conductor.manager.ConductorManager
topic=conductor
use_local=False
spice:
agent_enabled=True
enabled=False
html5proxy_base_url=http://10.33.8.112:6082/spice_auto.html
keymap=en-us
server_listen=127.0.0.1
server_proxyclient_address=127.0.0.1
========================================================================
==== Native threads ====
========================================================================
Thread ID 139989669103360
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989677496064
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990096934656
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990667343616
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990063363840
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990692521728
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990088541952
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990071756544
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990675736320
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990650558208
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990080149248
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990700914432
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990781155072
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989685888768
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989694281472
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990684129024
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139991167133504
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:81 in get_model
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:52 in create_report
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:42 in create
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:78 in dump
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:85 in __handle_signal
None
/usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py:61 in do_poll
return self.poll.poll(seconds)
/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py:84 in wait
presult = self.do_poll(seconds)
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:226 in run
self.wait(sleep_time)
Thread ID 139990658950912
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990789547776
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990046578432
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990054971136
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
========================================================================
==== Green threads ====
========================================================================
Thread ID 0
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/event.py:116 in wait
return hubs.get_hub().switch()
/home/berrange/src/cloud/nova/nova/utils.py:577 in wait
return self.done.wait()
/home/berrange/src/cloud/nova/nova/service.py:564 in wait
x.wait()
/home/berrange/src/cloud/nova/nova/service.py:148 in run_server
server.wait()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 1
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/green/select.py:70 in select
return hub.switch()
/usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
ready, _, _ = select([self], [], [], timeout)
/usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
sw.wait(timeout)
/usr/lib/python2.7/site-packages/qpid/concurrency.py:57 in wait
self.condition.wait(3)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
return self._waiter.wait(predicate, timeout=timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
result = self._wait(lambda: self.error or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
result = self.connection._ewait(lambda: self.error or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
result = self._ewait(lambda: self.closed or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
if self._ecwait(lambda: self.incoming, timeout):
<string>:6 in next_receiver
None
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:426 in _consume
nxt_receiver = self.session.next_receiver(timeout=timeout)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:375 in ensure
return method(*args, **kwargs)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:435 in iterconsume
yield self.ensure(_error_callback, _consume)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:518 in consume
it.next()
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:526 in _consumer_thread
self.consume()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 2
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
hub.switch()
/home/berrange/src/cloud/nova/nova/utils.py:638 in _inner
greenthread.sleep(idle)
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 3
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
hub.switch()
/home/berrange/src/cloud/nova/nova/utils.py:596 in _inner
greenthread.sleep(interval)
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
========================================================================
==== Package ====
========================================================================
Vendor: OpenStack Foundation
Product: OpenStack Nova
Version: 2013.1
========================================================================
