libera/#infobot/ Tuesday, 2018-09-04

DocScrutinizer05TimRiker: Hi!18:39
DocScrutinizer05:-D18:39
DocScrutinizer05TimRiker: you received the email I've tried to send to you?18:39
TimRikerI was out of town for the weekend. Looking into the bot now.18:41
DocScrutinizer05:-)18:41
DocScrutinizer05[2018 M08 30, Thu] [06:47:33 CEST]|Quit| infobot (ibot@208.53.50.136) has left this server (Ping timeout: 245 seconds).18:44
DocScrutinizer05last sign of life18:44
DocScrutinizer05or death ;-)18:44
DocScrutinizer05\o/ it's back18:44
DocScrutinizer05~wb18:46
infobotthx18:46
TimRikerlogs show some "Out of memory!" errors. huh? that confuses me. :)18:48
DocScrutinizer05eeek18:49
DocScrutinizer05do those logs also quote the amount of memory the bot used?18:50
TimRikerI'm not currently logging memory usage.18:50
DocScrutinizer05possibly the culprit been completely different process?18:50
TimRikerthe log entry is from the loop script. I don't think it's the script. it's tiny.18:51
DocScrutinizer05poor bot only suffering from its "flat" getting smaller and smaller18:51
TimRiker~die18:51
TimRiker~quit18:51
DocScrutinizer05o.O18:51
DocScrutinizer05wait. the loop script of bot, which I can "reboot"?18:53
TimRikeryes. it restarts the bot when the bot exits.18:53
DocScrutinizer05my keep-alove monitor had a runaway possibly sending a ssh ibot@eikers.org every minute for several hours18:53
DocScrutinizer05I hope I didn't trihher this issue18:54
DocScrutinizer05I encountered a strange ssh lockup as well, where the ssh session didn't complete but also wasn't interrupt-able by any means except SIGKILL18:55
DocScrutinizer05see attachment of mail I sent18:55
TimRikerit's possible. No worries. I don't see a cause yet. still looking through logs18:55
TimRikerI have upgraded the server from ubuntu 16.04 to 18.0418:56
DocScrutinizer05could you augment the loop script resp my ssh ibot@rikers command in a way so it echoes some useful diagnostics like `gree` and/or iptraf and/or dunnowhat before it closes the session with bot restart?18:58
DocScrutinizer05`free`18:58
DocScrutinizer05ps -C ibot18:59
DocScrutinizer05ifconfig18:59
DocScrutinizer05I'm not too savvy in spotting the nice informative /proc/`pidof ibot`/* nodes19:00
DocScrutinizer05I got no idea how sensitive the bot log itself might be. Maybe a `tail -n 60 ibot.log` would be very useful to evaluate _why_ the bot needs a reboot19:04
TimRikerok, adding some info. new command is $HOME/bin/restartbot19:08
DocScrutinizer05ooh, one of the last things I seen been trying to the bot: the d* commands for debian repo search etc19:09
DocScrutinizer05maybe this eats quite some memory?19:09
DocScrutinizer05~+help dfind19:10
infobot   Desc: Debian Packages (fallback to Contents) search19:10
infobot  Usage: dfind <string> [dist]19:10
infobotExample: dfind strings.h19:10
infobotExample: dfind x11amp potato19:10
DocScrutinizer05~+dfind systemd stable19:10
infobotUpdating debian files... please wait.19:10
infobotfailed.19:10
TimRikerdunno. it eats more than other commands. I think it's a forked command though, so it should not kill the bot, just the forked request might run out of memory.19:10
DocScrutinizer05apt I suppose, yes19:11
TimRiker~+dfind systemd stable19:11
infobotUpdating debian files... please wait.19:11
infobotfailed.19:11
DocScrutinizer05bitrotten anyway, I think19:11
DocScrutinizer05didn't manage to get a single useful result out of it19:11
DocScrutinizer05~+dfind systemd sid19:12
infobotUpdating debian files... please wait.19:12
infobotfailed.19:12
TimRikerit looks like it's pulling the files ok.19:12
DocScrutinizer05too fast19:13
DocScrutinizer05unless cache up to date19:13
DocScrutinizer05~+dfind x11amp potato19:14
infobotUpdating debian files... please wait.19:14
infobotfailed.19:14
DocScrutinizer05~+dfind x11amp19:14
infobotUpdating debian files... please wait.19:14
infobotfailed.19:14
DocScrutinizer05this one supposed to work, did last time:19:15
DocScrutinizer05~+dstats stable19:15
infobotUpdating debian files... please wait.19:15
infobotDebian(iS): internal error.19:15
DocScrutinizer05~+dstats sid19:15
infobotDebian Distro Stats on sid... 60612 packages, 2448 maintainers, 302568 MB installed size, 79363 MB compressed size.19:15
DocScrutinizer05ooops19:17
TimRiker~+dfind systemd stable19:20
infobotUpdating debian files... please wait.19:20
infobotDebian Package Listing of 'systemd' (13 of 26): charon-systemd ;; dh-systemd ;; golang-github-coreos-go-systemd-dev ;; kde-config-systemd ;; libconfig-model-systemd-perl ;; libnss-systemd ;; libpam-systemd ;; libsystemd-dev ;; libsystemd0 ;; live-config-systemd ;; lua-systemd ;; lua-systemd-dev ;; openvpn-systemd-resolved.19:20
DocScrutinizer05DUH! wow!19:20
TimRiker~+dstats stable19:21
infobotDebian Distro Stats on stretch... 51439 packages, 2324 maintainers, 215590 MB installed size, 60664 MB compressed size.19:21
TimRiker~+dstats testing19:21
infobotUpdating debian files... please wait.19:21
infobotDebian Distro Stats on buster... 55650 packages, 2330 maintainers, 266185 MB installed size, 70461 MB compressed size.19:21
TimRikerupdated the distro pointers, and pulling from us instead of nl.19:22
DocScrutinizer05:-)19:22
TimRiker~+dfind bzflag stable19:22
infobotDebian Package Listing of 'bzflag' (4): bzflag ;; bzflag-client ;; bzflag-data ;; bzflag-server.19:22
DocScrutinizer05~dauthor Riker19:23
infobotDebian Author Search of 'Riker' returned no results.19:23
DocScrutinizer05~dauthor Hartman19:23
infobotDebian Author Search of 'Hartman' (2): Sam Hartman ;; Richard Hartmann.19:23
TimRikerI'm probably not listed as the BZFlag author anymore. It's been a while since I've been active.19:24
TimRiker~dinfo bzflag19:24
infobotbzflag: (3D first person tank battle game), section games, is optional. Version: 2.4.14-1 (sid), Packaged size: 60 kB, Installed size: 94 kB19:24
TimRiker~dinfo bzflag stable19:24
infobotbzflag: (3D first person tank battle game), section games, is optional. Version: 2.4.8-1 (stretch), Packaged size: 54 kB, Installed size: 74 kB19:25
TimRiker~+dstats etch19:25
infobotUpdating debian files... please wait.19:25
infobotDebian(iS): internal error.19:25
DocScrutinizer05let me give my reboot mechanism a try, ok?19:26
TimRikerok19:26
TimRikeroh, wait...19:26
DocScrutinizer05debug2: channel 0: send open19:26
DocScrutinizer05debug1: Requesting no-more-sessions@openssh.com19:26
DocScrutinizer05debug1: Entering interactive session.19:26
DocScrutinizer05debug1: pledge: network19:26
DocScrutinizer05...19:26
DocScrutinizer05didumdidum19:27
TimRikerI didn't change the auth keys entry. try again?19:27
DocScrutinizer05first need to kill this session19:27
DocScrutinizer05hope I did19:27
DocScrutinizer05same...19:28
DocScrutinizer05hangs on debug1: pledge: network19:28
DocScrutinizer05friggin OpenSSH19:28
DocScrutinizer05the OpenSSH folks messed something up thoroughly lately19:29
TimRikerare you trying ipv6 or ipv4?19:29
TimRiker~chaninf19:30
TimRiker~chaninfo19:30
infobot117/119 channels, 3555 users, 2761 unique: #debian/1062, #kde/734, #devuan/252, #asterisk/167, #maemo/134, #gsoc/124, #oe/91, #neo900/82, #utah/53, #arm-netbook/42, #harmattan/42, #openmoko/40, #asterisk-dev/39, #qi-hardware/36, #cubieboard/34, #bzflag/34, #maemo-ssu/33, #openmoko-cdevel/33, #elinux/27, #wowhead/24, #curseforge/23, #debianppc/20, #webos-internals/15, #meego/15, #brlcad/15, #bzflag-chat/13, #uphpu/13, ##leaguesunited/12, ...19:30
DocScrutinizer05umm, locally IPv4, though via a DS-lite tunnel (IPv6) to a cgNAT where I got a fixed publicly visible IP19:30
DocScrutinizer05public IPv419:31
* DocScrutinizer05 waves19:31
TimRikerthat was me. testing. :)19:31
TimRikertrying to get to ssl now...19:33
DocScrutinizer05I gonna failover to my strict IPv4 uplink, as I've seen similar lockups with ssh and SSL a while ago on my cable DS-lite crap19:35
* DocScrutinizer05 pulls a RJ4519:35
TimRikerok. give me a bit, I'm working on ssl setup..19:35
DocScrutinizer05failover failed. NIIICEĀ”19:42
DocScrutinizer05backup link failed the very moment it been needed19:42
DocScrutinizer05https://pastebin.com/UEt0p8QZ19:43
DocScrutinizer05o.O19:49
TimRikerhmm. bot logs in, but does not go on...19:49
DocScrutinizer05I see19:49
DocScrutinizer05[2018-09-04 19:50:41] [Notify] purl is online (kornbluth.freenode.net).19:51
DocScrutinizer05~wb20:06
infobotthx20:06
DocScrutinizer05~+uptime20:07
infobot- Uptime for purl -20:07
infobotNow: 2m 36s running infobot 1.5.4 (SVN) -- linux20:07
infobot1: 59d 8h 41m 19s running infobot 1.5.4 (SVN) -- linux, ended Sun Nov 14 18:39:57 201020:08
infobot2: 57d 3h 9m 23s running infobot 1.5.4 (SVN) -- linux, ended Fri Jun 26 20:39:27 200920:08
infobot3: 53d 8h 24m 4s running infobot 1.5.4 (SVN) -- linux, ended Tue Nov  7 05:34:20 201720:08
DocScrutinizer05~+status20:08
infobotSince Tue Sep  4 18:05:23 2018, there have been 0 modifications, 1 question, 0 dunnos, 0 morons and 1 command.  I have been awake for 2m 59s this session, and currently reference 119767 factoids.  I'm using about 80032 kB of memory. With 0 active forks. Process time user/system 1.59/0.15 child 0/020:08
DocScrutinizer05why did she ignoe your ~die ?20:09
DocScrutinizer05ignore*20:09
DocScrutinizer05~+help die20:09
infobotno help on die.  Use 'help' without arguments.20:09
DocScrutinizer05hmmm20:09
TimRikercan't see why ssl is not working. It connects, logs in, but then hangs.20:10
DocScrutinizer05I'd bet this once been a command overleyed by a factoid, for those lacking the permissions20:10
DocScrutinizer05yeah, refer to my ssh session lockup20:11
DocScrutinizer05seems to me like a OpenSSH compatibility problem20:11
DocScrutinizer05~literal die20:13
infobot"die" is "<action> takes two shots to the head and crumples to the ground, lifeless."20:13
DocScrutinizer05so much for the factoid. I'm not sure the command ever been listed in ~help20:14
DocScrutinizer05[Sun 06 November 2016] [22:27:36] <DocScrutinizer05> die20:19
DocScrutinizer05[Sun 06 November 2016] [22:27:41] Quit infobot (ibot@rikers.org) has left this server (Quit: cyal8r).20:19
DocScrutinizer05TimRiker: should I try ssh login reboot again?20:20
DocScrutinizer05maybe the sshd log on your end reveals something?20:21
TimRiker~chandel #wowace20:22
infobotPrefix channel with '-' to delete for sure.20:22
TimRiker~chandel -#wowace20:22
infobotDeleting channel #wowace for sure!20:22
infobotLeaving #wowace...20:22
TimRiker~save20:23
infobotsaved user and chan files20:23
TimRikersure. go ahead and try it out. :)20:23
DocScrutinizer05reboot?`20:23
DocScrutinizer05ok20:23
DocScrutinizer05MHMMMM20:24
DocScrutinizer05TimRiker: https://pastebin.com/XFKZUuCj20:25
TimRikerhmm. command not found. full pathing...20:26
TimRikertry again?20:27
DocScrutinizer05at least it *works* now \o/20:27
DocScrutinizer05sure20:27
DocScrutinizer05GREAT!20:27
DocScrutinizer05TimRiker: excellent diagnostics, just need to get rid of my own crap in between now, by removing the "-vvv" parameter to ssh, and keeping this diagnostics persistent ( ssh ... >>ibot-reboot.log)20:31
DocScrutinizer05aaaawww20:31
DocScrutinizer05:-)20:32
TimRikerrestarted to disable debug.20:35
TimRiker~chaninfo20:35
infobot118/118 channels, 3649 users, 2830 unique: #debian/1055, #kde/735, #devuan/251, #asterisk/171, #maemo/134, #gsoc/124, #wowuidev/97, #oe/91, #neo900/82, #utah/53, #arm-netbook/43, #harmattan/42, #asterisk-dev/40, #openmoko/40, #qi-hardware/36, #cubieboard/35, #bzflag/34, #maemo-ssu/33, #openmoko-cdevel/32, #elinux/26, #wowhead/25, #curseforge/22, #debianppc/20, #meego/15, #brlcad/15, #webos-internals/15, #uphpu/13, #bzflag-chat/13, ...20:35
TimRikerremoved from #wowace as they have ssl required, and alas, I can't get infobot working over ssl.20:36
DocScrutinizer05aaah, disable debug. Yeah, I was about to say:20:39
DocScrutinizer05HMMMM20:39
DocScrutinizer0518:27.32 !DEBUG! main::IsNickInChan[1035] iNIC: hash channels contains mixed cased chan!!!20:39
DocScrutinizer0518:27.32 LOG: last message repeated 233 times20:39
DocScrutinizer05TimRiker: users love it :-)  http://maemo.cloud-7.de/irclogs/freenode/_devuan/latest.log.html#t2018-09-04T19:21:5920:43
DocScrutinizer05now if only we could solve the mystery why the bot is doing a restart each night 00:18 UTC20:47
TimRikerindeed. :)20:47
DocScrutinizer05next time I see her vanish, I might fire up a ssh reboot command and have a look at the diagnostics, maybe the ps gives away something useful20:48
* DocScrutinizer05 glares at logrotate20:48
DocScrutinizer05and cron-daily20:48
DocScrutinizer05do the bot logs start at 00:21 UTC by any chance?20:51
DocScrutinizer05and previous rotate-saved one ends at 00:18?20:52
DocScrutinizer05I'd not feel too surprised if there was a ibot command "reopen-logfiles" and systemd-crond never heard of this command and instead restarts the bot like it does with every generic service on logrotate20:57
DocScrutinizer05loosely on topic:  https://dev1galaxy.org/viewtopic.php?id=234520:58
DocScrutinizer05TimRiker: looking at diagnostics, I wonder if you do pstree *after* killall ibot21:06
TimRikerok21:07
DocScrutinizer05ok?21:08
TimRikerchanged. pstree before and after kill now.21:08
DocScrutinizer05is this what you do, and do you think it's ok? or do you swap the sequence?21:09
DocScrutinizer05aah :-)21:09
TimRikerit should show the loop in a sleep 10 after the kill. might not have killed it yet though, might still be shutting down.21:09
DocScrutinizer05cool21:09
DocScrutinizer05ps before reboot for sure is more interesting than after, at least for diagnosing the reason why it had to get rebooted21:10
DocScrutinizer05also CPU percent, used network bandwidth up/down during last few seconds21:13
TimRikerI added uptime there. thoughts on a good bandwidth command?21:15
TimRikerremember the server has both ipv4 and ipv6 interfaces.21:15
DocScrutinizer05no idea, seems tricky though *should* be straight simple21:18
* DocScrutinizer05 browses through all /proc/<PID>/*21:18
DocScrutinizer05TimRiker: [quote]  NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process.21:23
DocScrutinizer05https://pastebin.com/Zq8kaVBN nethogs --help21:25
DocScrutinizer05interactive though https://i.imgur.com/PUTbtKt.png -c 1 results in useless empty display21:28
DocScrutinizer05TimRiker: nethogs -t21:34
DocScrutinizer05TimRiker: nethogs -t -c 2021:34
TimRikerneeds root, and I don't see a way to show only the current process.21:36
TimRikeradded a netstat -ntp|grep perl21:39
DocScrutinizer05yep, prolly both points are not solvable21:48
DocScrutinizer05the second maybe with a grep21:49
DocScrutinizer05looking at /proc/self/net/stat/nf_conntrack it's quite some gibberish ;-)21:51
DocScrutinizer05no idea if huimans can decode /proc/self/net/dev and if it has useful info21:52
DocScrutinizer05as long as your system stillprovides netstat :-/  I hate it when they phase out old reliable excellent tools in favor of new "do it all" monsters21:59
DocScrutinizer05https://dougvitale.wordpress.com/2011/12/21/deprecated-linux-networking-commands-and-their-replacements/22:01
DocScrutinizer05>:-(22:01
DocScrutinizer05gotta love the "Replacement: Not apparent"22:07

Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!