For bugs enter the bug severity level. Do not set any labels.
There are a set of tests that intermittently fail in nightly due to an assertion error. I would guess that the failure rate is roughly 10-20% of the time. The tests are in api/test_pools.py. Both occasionally fail. I've reproduced the error with some debug information that is shown below. We delete the shared pool and then expect the loadbalancer status to go 'ACTIVE' once the agent communicates to the driver that the operation is complete. The test checks for the ACTIVE loadbalancer, and then retrieves the service object from the driver directly. Sometimes that service object has the pool still in PENDING_DELETE state. This causes the test to fail. I've also traced the logs and the right things appear to be happening in the agent and neutron. Both of the operations of setting the lb status and deleting the pool are performed within a database transaction.
Here's the debug service object output. The steps are delete the pool, check the lb status for up to 1 minute until it goes ACTIVE, then check the service object:
2017-08-04 20:43:54.478 6280 INFO tempest.lib.common.rest_client [req-28f56e61-ce42-4f28-a876-d2396995658d - - - - -] Request (PoolTestJSON:test_create_shared_detached_pools): 204 DELETE http://10.190.4.199:9696/v2.0/lbaas/pools/64c6e329-c649-4da7-9485-e012e9d4e5c9 0.824s
2017-08-04 20:43:54.654 6280 INFO tempest.lib.common.rest_client [req-28f56e61-ce42-4f28-a876-d2396995658d - - - - -] Request (PoolTestJSON:test_create_shared_detached_pools): 200 GET http://10.190.4.199:9696/v2.0/lbaas/loadbalancers/4b1458cb-d965-4194-a4f8-7e9b2c176b7f 0.175s
2017-08-04 20:43:56.038 6280 INFO tempest.lib.common.rest_client [req-28f56e61-ce42-4f28-a876-d2396995658d - - - - -] Request (PoolTestJSON:test_create_shared_detached_pools): 200 GET http://10.190.4.199:9696/v2.0/lbaas/loadbalancers/4b1458cb-d965-4194-a4f8-7e9b2c176b7f 0.381s
{ u'healthmonitors': [],
u'l7policies': [],
u'l7policy_rules': [],
u'listeners': [ { u'admin_state_up': True,
u'connection_limit': -1,
u'default_pool_id': u'64c6e329-c649-4da7-9485-e012e9d4e5c9',
u'default_tls_container_id': None,
u'description': u'First Listener',
u'id': u'7d9cc438-65ed-4628-b73e-5b39a5c97aa8',
u'l7_policies': [],
u'loadbalancer_id': u'4b1458cb-d965-4194-a4f8-7e9b2c176b7f',
u'name': u'',
u'operating_status': u'ONLINE',
u'protocol': u'HTTP',
u'protocol_port': 80,
u'provisioning_status': u'ACTIVE',
u'sni_containers': [],
u'tenant_id': u'4796cd52685841639a64169222f68eda'},
{ u'admin_state_up': True,
u'connection_limit': -1,
u'default_pool_id': u'64c6e329-c649-4da7-9485-e012e9d4e5c9',
u'default_tls_container_id': None,
u'description': u'Second Listener',
u'id': u'a8af0936-2b0e-49f7-a8c4-394cb7c934ae',
u'l7_policies': [],
u'loadbalancer_id': u'4b1458cb-d965-4194-a4f8-7e9b2c176b7f',
u'name': u'',
u'operating_status': u'ONLINE',
u'protocol': u'HTTP',
u'protocol_port': 8080,
u'provisioning_status': u'ACTIVE',
u'sni_containers': [],
u'tenant_id': u'4796cd52685841639a64169222f68eda'}],
u'loadbalancer': { u'admin_state_up': True,
u'description': u'',
u'gre_vteps': [],
u'id': u'4b1458cb-d965-4194-a4f8-7e9b2c176b7f',
u'listeners': [ { u'id': u'7d9cc438-65ed-4628-b73e-5b39a5c97aa8'},
{ u'id': u'a8af0936-2b0e-49f7-a8c4-394cb7c934ae'}],
u'name': u'',
u'network_id': u'914a399a-280a-44cf-86be-9135b2bb57a5',
u'operating_status': u'ONLINE',
u'pools': [ { u'id': u'64c6e329-c649-4da7-9485-e012e9d4e5c9'}],
u'provider': u'f5networks',
u'provisioning_status': u'ACTIVE',
u'tenant_id': u'4796cd52685841639a64169222f68eda',
u'vip_address': u'10.2.4.3',
u'vip_port': { u'admin_state_up': True,
u'allowed_address_pairs': [],
u'binding:host_id': u'host-34.int.lineratesystems.com:7bb1137f-5c7b-5b0c-8d79-ed726b66fb0a',
u'binding:profile': { },
u'binding:vif_details': { },
u'binding:vif_type': u'binding_failed',
u'binding:vnic_type': u'normal',
u'created_at': u'2017-08-04T20:43:24',
u'description': None,
u'device_id': u'fe1d9deb-f8a1-5651-ad5d-06a9f05ddd38',
u'device_owner': u'network:f5lbaasv2',
u'dns_name': None,
u'extra_dhcp_opts': [],
u'fixed_ips': [ { u'ip_address': u'10.2.4.3',
u'subnet_id': u'ca8da2f2-00f7-4fe6-b04d-eba5432e1944'}],
u'id': u'b017b80c-6207-4d22-a3f7-f28933734638',
u'mac_address': u'fa:16:3e:4d:02:a3',
u'name': u'loadbalancer-4b1458cb-d965-4194-a4f8-7e9b2c176b7f',
u'network_id': u'914a399a-280a-44cf-86be-9135b2bb57a5',
u'security_groups': [ u'f271367b-ad7e-433e-ac3a-b592b175d613'],
u'status': u'DOWN',
u'tenant_id': u'4796cd52685841639a64169222f68eda',
u'updated_at': u'2017-08-04T20:43:24'},
u'vip_port_id': u'b017b80c-6207-4d22-a3f7-f28933734638',
u'vip_subnet_id': u'ca8da2f2-00f7-4fe6-b04d-eba5432e1944',
u'vxlan_vteps': [ u'201.0.36.1',
u'201.0.38.10',
u'201.0.37.1',
u'201.0.35.1']},
u'members': [],
u'networks': { u'914a399a-280a-44cf-86be-9135b2bb57a5': { u'admin_state_up': True,
u'availability_zone_hints': [ ],
u'availability_zones': [ u'nova'],
u'created_at': u'2017-08-04T20:43:23',
u'description': u'',
u'id': u'914a399a-280a-44cf-86be-9135b2bb57a5',
u'ipv4_address_scope': None,
u'ipv6_address_scope': None,
u'mtu': 1450,
u'name': u'network-1637161001',
u'provider:network_type': u'vxlan',
u'provider:physical_network': None,
u'provider:segmentation_id': 59,
u'router:external': False,
u'shared': False,
u'status': u'ACTIVE',
u'subnets': [ u'ca8da2f2-00f7-4fe6-b04d-eba5432e1944'],
u'tags': [ ],
u'tenant_id': u'4796cd52685841639a64169222f68eda',
u'updated_at': u'2017-08-04T20:43:23',
u'vlan_transparent': None}},
u'pools': [ { u'admin_state_up': True,
u'description': u'Shared pool',
u'healthmonitor_id': None,
u'id': u'64c6e329-c649-4da7-9485-e012e9d4e5c9',
u'l7_policies': [],
u'lb_algorithm': u'ROUND_ROBIN',
u'loadbalancer_id': u'4b1458cb-d965-4194-a4f8-7e9b2c176b7f',
u'members': [],
u'name': u'',
u'operating_status': u'ONLINE',
u'protocol': u'HTTP',
u'provisioning_status': u'PENDING_DELETE',
u'sessionpersistence': None,
u'tenant_id': u'4796cd52685841639a64169222f68eda'}],
u'subnets': { u'ca8da2f2-00f7-4fe6-b04d-eba5432e1944': { u'allocation_pools': [ { u'end': u'10.2.4.14',
u'start': u'10.2.4.2'}],
u'cidr': u'10.2.4.0/28',
u'created_at': u'2017-08-04T20:43:23',
u'description': u'',
u'dns_nameservers': [ ],
u'enable_dhcp': True,
u'gateway_ip': u'10.2.4.1',
u'host_routes': [ ],
u'id': u'ca8da2f2-00f7-4fe6-b04d-eba5432e1944',
u'ip_version': 4,
u'ipv6_address_mode': None,
u'ipv6_ra_mode': None,
u'name': u'',
u'network_id': u'914a399a-280a-44cf-86be-9135b2bb57a5',
u'shared': False,
u'subnetpool_id': None,
u'tenant_id': u'4796cd52685841639a64169222f68eda',
u'updated_at': u'2017-08-04T20:43:23'}}}
OpenStack Release
mitaka
Bug Severity
For bugs enter the bug severity level. Do not set any labels.
Severity: <Fill in level: 1 through 5>
Severity level definitions:
Description
There are a set of tests that intermittently fail in nightly due to an assertion error. I would guess that the failure rate is roughly 10-20% of the time. The tests are in
api/test_pools.py. Both occasionally fail. I've reproduced the error with some debug information that is shown below. We delete the shared pool and then expect the loadbalancer status to go 'ACTIVE' once the agent communicates to the driver that the operation is complete. The test checks for the ACTIVE loadbalancer, and then retrieves the service object from the driver directly. Sometimes that service object has the pool still in PENDING_DELETE state. This causes the test to fail. I've also traced the logs and the right things appear to be happening in the agent and neutron. Both of the operations of setting the lb status and deleting the pool are performed within a database transaction.Here's the debug service object output. The steps are delete the pool, check the lb status for up to 1 minute until it goes ACTIVE, then check the service object: