From dkg at fifthhorseman.net Thu Jan 18 02:15:46 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Wed, 17 Jan 2018 21:15:46 -0500 Subject: [getdns-users] Context.timeout in python-getdns Message-ID: <87y3kvud2l.fsf@fifthhorseman.net> Hi getdns folks-- I'm using the python getdns bindings, and i've done something like following: import getdns ctx = getdns.Context() ctx.timeout = 15000 def callback(type, result, userarg, tid): pass def lookup_addresses(addresses): (addr, family) in addresses: q = { 'address_data': addr, 'address_type': 'IPv' + str(family) } ctx.hostname(q, userarg=addr, callback='callback') ctx.run() lookup_addresses([('192.0.2.5', 4), ('192.0.2.6', 4)]) The size and contents of the list passed to lookup_addresses() may vary, and i invoke lookup_addresses() more than once in the lifetime of the program. When i time the "lookup_addresses" function, it sometimes takes far more than 15 seconds to complete, despite having set ctx.timeout to 15000. and when i strace the process, i see some remarkable things, including invocations of poll() with a timeout variable far greater than 15000, like so: poll([{fd=5, events=POLLIN}, {fd=4, events=POLLIN}, {fd=-1}], 3, 67688) and 4 and 5 are both UDP ports, with no other getdns-controlled file descriptors open in the process. Any idea what's going on? --dkg PS: version information (debian packages on amd64): libevent-core-2.1-6 2.1.8-stable-4 libgetdns6 1.2.1-2 python3-getdns 1.0.0~b1-1+b2 From dkg at fifthhorseman.net Thu Jan 18 02:23:06 2018 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Wed, 17 Jan 2018 21:23:06 -0500 Subject: [getdns-users] Context.timeout in python-getdns In-Reply-To: <87y3kvud2l.fsf@fifthhorseman.net> References: <87y3kvud2l.fsf@fifthhorseman.net> Message-ID: <87vafzucqd.fsf@fifthhorseman.net> On Wed 2018-01-17 21:15:46 -0500, Daniel Kahn Gillmor wrote: > and when i strace the process, i see some remarkable things, including > invocations of poll() with a timeout variable far greater than 15000, > like so: > > poll([{fd=5, events=POLLIN}, {fd=4, events=POLLIN}, {fd=-1}], 3, 67688) below is a stream of strace output during a particularly long wait. Note the series of timeouts: 72 seconds, 30 seconds, 96 seconds, 30 seconds, and finally another 96 second timeout. all of this is within one iteration of lookup_addresses() :/ let me know if there's better debugging information you'd want to see... --dkg poll([{fd=4, events=POLLIN}], 1, 71704) = 0 (Timeout) <71.775838> socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 <0.000139> fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) <0.000064> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000107> connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("34.237.254.79")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000052> close(4) = 0 <0.000014> poll([{fd=5, events=POLLOUT}, {fd=-1}], 2, 29999) = 0 (Timeout) <30.029080> close(5) = 0 <0.000048> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 <0.000026> setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000011> bind(4, {sa_family=AF_INET, sin_port=htons(4459), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000021> fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) <0.000011> fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008> sendto(4, "\375F\0\20\0\1\0\0\0\0\0\1\3ns2\6ittltd\3com\0\0\34\0\1\0\0)\377\377\0\0\200\0\0\0", 43, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("52.207.176.29")}, 16) = 43 <0.000035> poll([{fd=4, events=POLLIN}], 1, 96255) = 0 (Timeout) <96.351372> socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 <0.000115> fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) <0.000070> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000039> connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("52.207.176.29")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000035> close(4) = 0 <0.000011> poll([{fd=5, events=POLLOUT}, {fd=-1}], 2, 29999) = 0 (Timeout) <30.029168> close(5) = 0 <0.000065> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 <0.000032> setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000071> bind(4, {sa_family=AF_INET, sin_port=htons(59558), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000017> fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) <0.000010> fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000010> sendto(4, "\225\260\0\20\0\1\0\0\0\0\0\1\4nsg2\rnamebrightdns\3com\0\0\34\0\1\0\0)\377\377\0\0\200\0\0\0", 51, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("54.208.25.173")}, 16) = 51 <0.000046> poll([{fd=4, events=POLLIN}], 1, 886) = 0 (Timeout) <0.886987> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 5 <0.000059> setsockopt(5, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000082> bind(5, {sa_family=AF_INET, sin_port=htons(51779), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000016> fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) <0.000011> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000011> sendto(5, "$\27\0\20\0\1\0\0\0\0\0\1\4nsg2\rnamebrightdns\3com\0\0\34\0\1\0\0)\5\300\0\0\200\0\0\0", 51, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("54.208.25.173")}, 16) = 51 <0.000044> close(4) = 0 <0.000019> poll([{fd=5, events=POLLIN}, {fd=-1}], 2, 886) = 1 ([{fd=5, revents=POLLIN}]) <0.815401> recvfrom(5, "$\27\204\0\0\1\0\0\0\0\0\1\4nsg2\rnamebrightdns\3com\0\0\34\0\1\4nsg2\rnamebrightdns\3com\0\0\1\0\1\0\0\0\0\0\4\"\355\376O\0\0)\5\300\0\0\200\0\0\0", 65552, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("54.208.25.173")}, [128->16]) = 89 <0.000036> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 <0.000026> setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000016> bind(4, {sa_family=AF_INET, sin_port=htons(8014), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000011> fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) <0.000009> fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000012> sendto(4, "\1p\0\20\0\1\0\0\0\0\0\1\4nsg1\rnamebrightdns\3com\0\0\34\0\1\0\0)\377\377\0\0\200\0\0\0", 51, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("34.194.232.55")}, 16) = 51 <0.000030> close(5) = 0 <0.000016> recvfrom(4, 0x561a85269e90, 65552, 0, 0x7ffcae1a5738, [128]) = -1 EAGAIN (Resource temporarily unavailable) <0.000022> poll([{fd=4, events=POLLIN}], 1, 1503) = 1 ([{fd=4, revents=POLLIN}]) <1.009433> recvfrom(4, "\1p\204\0\0\1\0\0\0\0\0\1\4nsg1\rnamebrightdns\3com\0\0\34\0\1\4nsg1\rnamebrightdns\3com\0\0\1\0\1\0\0\0\0\0\4\"\341&\272\0\0)\377\377\0\0\200\0\0\0", 65552, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("34.194.232.55")}, [128->16]) = 89 <0.000047> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 5 <0.000029> setsockopt(5, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000019> bind(5, {sa_family=AF_INET, sin_port=htons(32857), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000022> fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) <0.000009> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000009> sendto(5, "Ty\0\20\0\1\0\0\0\0\0\1\3ns1\6ittltd\3com\0\0\34\0\1\0\0)\377\377\0\0\200\0\0\0", 43, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("52.203.116.207")}, 16) = 43 <0.000033> socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 6 <0.000015> setsockopt(6, SOL_IP, IP_MTU_DISCOVER, [5], 4) = 0 <0.000010> bind(6, {sa_family=AF_INET, sin_port=htons(28285), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 <0.000012> fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) <0.000009> fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000009> sendto(6, "\260\212\0\20\0\1\0\0\0\0\0\1\3ns1\6ittltd\3com\0\0\1\0\1\0\0)\377\377\0\0\200\0\0\0", 43, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("34.225.38.186")}, 16) = 43 <0.000023> close(4) = 0 <0.000017> recvfrom(5, 0x561a85269e90, 65552, 0, 0x7ffcae1a5738, [128]) = -1 EAGAIN (Resource temporarily unavailable) <0.000009> recvfrom(6, 0x561a85269e90, 65552, 0, 0x7ffcae1a5738, [128]) = -1 EAGAIN (Resource temporarily unavailable) <0.000010> poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, 96255 From melinda.shore at nomountain.net Fri Jan 19 06:31:12 2018 From: melinda.shore at nomountain.net (Melinda Shore) Date: Thu, 18 Jan 2018 21:31:12 -0900 Subject: [getdns-users] Context.timeout in python-getdns In-Reply-To: <87y3kvud2l.fsf@fifthhorseman.net> References: <87y3kvud2l.fsf@fifthhorseman.net> Message-ID: <17ce1c53-41c6-285a-ef59-4c9a91b62d06@nomountain.net> Hi, Daniel: Thanks for the very clear description of what you're seeing. The Python bindings basically wrap libgetdns and handle data type conversions in both directions and don't do much more than that. I have some thoughts about what might be causing this - I'll definitely take a look at it this weekend. Thanks again! Melinda -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 874 bytes Desc: OpenPGP digital signature URL: From melinda.shore at nomountain.net Mon Jan 22 03:02:46 2018 From: melinda.shore at nomountain.net (Melinda Shore) Date: Sun, 21 Jan 2018 18:02:46 -0900 Subject: [getdns-users] Context.timeout in python-getdns In-Reply-To: <87vafzucqd.fsf@fifthhorseman.net> References: <87y3kvud2l.fsf@fifthhorseman.net> <87vafzucqd.fsf@fifthhorseman.net> Message-ID: Hi, Daniel: I am unable to reproduce this - I am not seeing unexpected values being used by poll() as its timeout value. The environment in which I tested it was Ubuntu 16.04, libgetdns version 1.3.0 (getdns.Context().version_string) and getdns Python module version v1.0.0b (getdns.__version__). Best guess would be that the bindings are loading a different libgetdns from the one you compiled against. If you can let me know what your runtime environment is I can try to duplicate it here. Thanks! Melinda -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 874 bytes Desc: OpenPGP digital signature URL: From paul at nohats.ca Tue Jan 2 18:12:28 2018 From: paul at nohats.ca (Paul Wouters) Date: Tue, 2 Jan 2018 13:12:28 -0500 (EST) Subject: [getdns-api] getdns-1.3.0 released In-Reply-To: <6ca5e341-f8fd-1f5f-887e-1262c96c66eb@nlnetlabs.nl> References: <6ca5e341-f8fd-1f5f-887e-1262c96c66eb@nlnetlabs.nl> Message-ID: On Fri, 22 Dec 2017, Willem Toorop wrote: > I am pleased to announce the new Christmas 2017 bugfix release, version > 1.3.0 of getdns. I've pushed this into fedora rawhide. Note I filed a bug: https://github.com/getdnsapi/stubby/issues/62 which relates to systemd/tmpfiles use. And stubby also does not seem to fail to start properly when unbound is already running on port 53. Note that stubby could also use a man page :) Paul