DocScrutinizer05 | TimRiker: Hi! | 18:39 |
---|---|---|
DocScrutinizer05 | :-D | 18:39 |
DocScrutinizer05 | TimRiker: you received the email I've tried to send to you? | 18:39 |
TimRiker | I was out of town for the weekend. Looking into the bot now. | 18:41 |
DocScrutinizer05 | :-) | 18:41 |
DocScrutinizer05 | [2018 M08 30, Thu] [06:47:33 CEST]|Quit| infobot (ibot@208.53.50.136) has left this server (Ping timeout: 245 seconds). | 18:44 |
DocScrutinizer05 | last sign of life | 18:44 |
DocScrutinizer05 | or death ;-) | 18:44 |
DocScrutinizer05 | \o/ it's back | 18:44 |
DocScrutinizer05 | ~wb | 18:46 |
infobot | thx | 18:46 |
TimRiker | logs show some "Out of memory!" errors. huh? that confuses me. :) | 18:48 |
DocScrutinizer05 | eeek | 18:49 |
DocScrutinizer05 | do those logs also quote the amount of memory the bot used? | 18:50 |
TimRiker | I'm not currently logging memory usage. | 18:50 |
DocScrutinizer05 | possibly the culprit been completely different process? | 18:50 |
TimRiker | the log entry is from the loop script. I don't think it's the script. it's tiny. | 18:51 |
DocScrutinizer05 | poor bot only suffering from its "flat" getting smaller and smaller | 18:51 |
TimRiker | ~die | 18:51 |
TimRiker | ~quit | 18:51 |
DocScrutinizer05 | o.O | 18:51 |
DocScrutinizer05 | wait. the loop script of bot, which I can "reboot"? | 18:53 |
TimRiker | yes. it restarts the bot when the bot exits. | 18:53 |
DocScrutinizer05 | my keep-alove monitor had a runaway possibly sending a ssh ibot@eikers.org every minute for several hours | 18:53 |
DocScrutinizer05 | I hope I didn't trihher this issue | 18:54 |
DocScrutinizer05 | I encountered a strange ssh lockup as well, where the ssh session didn't complete but also wasn't interrupt-able by any means except SIGKILL | 18:55 |
DocScrutinizer05 | see attachment of mail I sent | 18:55 |
TimRiker | it's possible. No worries. I don't see a cause yet. still looking through logs | 18:55 |
TimRiker | I have upgraded the server from ubuntu 16.04 to 18.04 | 18:56 |
DocScrutinizer05 | could you augment the loop script resp my ssh ibot@rikers command in a way so it echoes some useful diagnostics like `gree` and/or iptraf and/or dunnowhat before it closes the session with bot restart? | 18:58 |
DocScrutinizer05 | `free` | 18:58 |
DocScrutinizer05 | ps -C ibot | 18:59 |
DocScrutinizer05 | ifconfig | 18:59 |
DocScrutinizer05 | I'm not too savvy in spotting the nice informative /proc/`pidof ibot`/* nodes | 19:00 |
DocScrutinizer05 | I got no idea how sensitive the bot log itself might be. Maybe a `tail -n 60 ibot.log` would be very useful to evaluate _why_ the bot needs a reboot | 19:04 |
TimRiker | ok, adding some info. new command is $HOME/bin/restartbot | 19:08 |
DocScrutinizer05 | ooh, one of the last things I seen been trying to the bot: the d* commands for debian repo search etc | 19:09 |
DocScrutinizer05 | maybe this eats quite some memory? | 19:09 |
DocScrutinizer05 | ~+help dfind | 19:10 |
infobot | Desc: Debian Packages (fallback to Contents) search | 19:10 |
infobot | Usage: dfind <string> [dist] | 19:10 |
infobot | Example: dfind strings.h | 19:10 |
infobot | Example: dfind x11amp potato | 19:10 |
DocScrutinizer05 | ~+dfind systemd stable | 19:10 |
infobot | Updating debian files... please wait. | 19:10 |
infobot | failed. | 19:10 |
TimRiker | dunno. it eats more than other commands. I think it's a forked command though, so it should not kill the bot, just the forked request might run out of memory. | 19:10 |
DocScrutinizer05 | apt I suppose, yes | 19:11 |
TimRiker | ~+dfind systemd stable | 19:11 |
infobot | Updating debian files... please wait. | 19:11 |
infobot | failed. | 19:11 |
DocScrutinizer05 | bitrotten anyway, I think | 19:11 |
DocScrutinizer05 | didn't manage to get a single useful result out of it | 19:11 |
DocScrutinizer05 | ~+dfind systemd sid | 19:12 |
infobot | Updating debian files... please wait. | 19:12 |
infobot | failed. | 19:12 |
TimRiker | it looks like it's pulling the files ok. | 19:12 |
DocScrutinizer05 | too fast | 19:13 |
DocScrutinizer05 | unless cache up to date | 19:13 |
DocScrutinizer05 | ~+dfind x11amp potato | 19:14 |
infobot | Updating debian files... please wait. | 19:14 |
infobot | failed. | 19:14 |
DocScrutinizer05 | ~+dfind x11amp | 19:14 |
infobot | Updating debian files... please wait. | 19:14 |
infobot | failed. | 19:14 |
DocScrutinizer05 | this one supposed to work, did last time: | 19:15 |
DocScrutinizer05 | ~+dstats stable | 19:15 |
infobot | Updating debian files... please wait. | 19:15 |
infobot | Debian(iS): internal error. | 19:15 |
DocScrutinizer05 | ~+dstats sid | 19:15 |
infobot | Debian Distro Stats on sid... 60612 packages, 2448 maintainers, 302568 MB installed size, 79363 MB compressed size. | 19:15 |
DocScrutinizer05 | ooops | 19:17 |
TimRiker | ~+dfind systemd stable | 19:20 |
infobot | Updating debian files... please wait. | 19:20 |
infobot | Debian Package Listing of 'systemd' (13 of 26): charon-systemd ;; dh-systemd ;; golang-github-coreos-go-systemd-dev ;; kde-config-systemd ;; libconfig-model-systemd-perl ;; libnss-systemd ;; libpam-systemd ;; libsystemd-dev ;; libsystemd0 ;; live-config-systemd ;; lua-systemd ;; lua-systemd-dev ;; openvpn-systemd-resolved. | 19:20 |
DocScrutinizer05 | DUH! wow! | 19:20 |
TimRiker | ~+dstats stable | 19:21 |
infobot | Debian Distro Stats on stretch... 51439 packages, 2324 maintainers, 215590 MB installed size, 60664 MB compressed size. | 19:21 |
TimRiker | ~+dstats testing | 19:21 |
infobot | Updating debian files... please wait. | 19:21 |
infobot | Debian Distro Stats on buster... 55650 packages, 2330 maintainers, 266185 MB installed size, 70461 MB compressed size. | 19:21 |
TimRiker | updated the distro pointers, and pulling from us instead of nl. | 19:22 |
DocScrutinizer05 | :-) | 19:22 |
TimRiker | ~+dfind bzflag stable | 19:22 |
infobot | Debian Package Listing of 'bzflag' (4): bzflag ;; bzflag-client ;; bzflag-data ;; bzflag-server. | 19:22 |
DocScrutinizer05 | ~dauthor Riker | 19:23 |
infobot | Debian Author Search of 'Riker' returned no results. | 19:23 |
DocScrutinizer05 | ~dauthor Hartman | 19:23 |
infobot | Debian Author Search of 'Hartman' (2): Sam Hartman ;; Richard Hartmann. | 19:23 |
TimRiker | I'm probably not listed as the BZFlag author anymore. It's been a while since I've been active. | 19:24 |
TimRiker | ~dinfo bzflag | 19:24 |
infobot | bzflag: (3D first person tank battle game), section games, is optional. Version: 2.4.14-1 (sid), Packaged size: 60 kB, Installed size: 94 kB | 19:24 |
TimRiker | ~dinfo bzflag stable | 19:24 |
infobot | bzflag: (3D first person tank battle game), section games, is optional. Version: 2.4.8-1 (stretch), Packaged size: 54 kB, Installed size: 74 kB | 19:25 |
TimRiker | ~+dstats etch | 19:25 |
infobot | Updating debian files... please wait. | 19:25 |
infobot | Debian(iS): internal error. | 19:25 |
DocScrutinizer05 | let me give my reboot mechanism a try, ok? | 19:26 |
TimRiker | ok | 19:26 |
TimRiker | oh, wait... | 19:26 |
DocScrutinizer05 | debug2: channel 0: send open | 19:26 |
DocScrutinizer05 | debug1: Requesting no-more-sessions@openssh.com | 19:26 |
DocScrutinizer05 | debug1: Entering interactive session. | 19:26 |
DocScrutinizer05 | debug1: pledge: network | 19:26 |
DocScrutinizer05 | ... | 19:26 |
DocScrutinizer05 | didumdidum | 19:27 |
TimRiker | I didn't change the auth keys entry. try again? | 19:27 |
DocScrutinizer05 | first need to kill this session | 19:27 |
DocScrutinizer05 | hope I did | 19:27 |
DocScrutinizer05 | same... | 19:28 |
DocScrutinizer05 | hangs on debug1: pledge: network | 19:28 |
DocScrutinizer05 | friggin OpenSSH | 19:28 |
DocScrutinizer05 | the OpenSSH folks messed something up thoroughly lately | 19:29 |
TimRiker | are you trying ipv6 or ipv4? | 19:29 |
TimRiker | ~chaninf | 19:30 |
TimRiker | ~chaninfo | 19:30 |
infobot | 117/119 channels, 3555 users, 2761 unique: #debian/1062, #kde/734, #devuan/252, #asterisk/167, #maemo/134, #gsoc/124, #oe/91, #neo900/82, #utah/53, #arm-netbook/42, #harmattan/42, #openmoko/40, #asterisk-dev/39, #qi-hardware/36, #cubieboard/34, #bzflag/34, #maemo-ssu/33, #openmoko-cdevel/33, #elinux/27, #wowhead/24, #curseforge/23, #debianppc/20, #webos-internals/15, #meego/15, #brlcad/15, #bzflag-chat/13, #uphpu/13, ##leaguesunited/12, ... | 19:30 |
DocScrutinizer05 | umm, locally IPv4, though via a DS-lite tunnel (IPv6) to a cgNAT where I got a fixed publicly visible IP | 19:30 |
DocScrutinizer05 | public IPv4 | 19:31 |
* DocScrutinizer05 waves | 19:31 | |
TimRiker | that was me. testing. :) | 19:31 |
TimRiker | trying to get to ssl now... | 19:33 |
DocScrutinizer05 | I gonna failover to my strict IPv4 uplink, as I've seen similar lockups with ssh and SSL a while ago on my cable DS-lite crap | 19:35 |
* DocScrutinizer05 pulls a RJ45 | 19:35 | |
TimRiker | ok. give me a bit, I'm working on ssl setup.. | 19:35 |
DocScrutinizer05 | failover failed. NIIICEĀ” | 19:42 |
DocScrutinizer05 | backup link failed the very moment it been needed | 19:42 |
DocScrutinizer05 | https://pastebin.com/UEt0p8QZ | 19:43 |
DocScrutinizer05 | o.O | 19:49 |
TimRiker | hmm. bot logs in, but does not go on... | 19:49 |
DocScrutinizer05 | I see | 19:49 |
DocScrutinizer05 | [2018-09-04 19:50:41] [Notify] purl is online (kornbluth.freenode.net). | 19:51 |
DocScrutinizer05 | ~wb | 20:06 |
infobot | thx | 20:06 |
DocScrutinizer05 | ~+uptime | 20:07 |
infobot | - Uptime for purl - | 20:07 |
infobot | Now: 2m 36s running infobot 1.5.4 (SVN) -- linux | 20:07 |
infobot | 1: 59d 8h 41m 19s running infobot 1.5.4 (SVN) -- linux, ended Sun Nov 14 18:39:57 2010 | 20:08 |
infobot | 2: 57d 3h 9m 23s running infobot 1.5.4 (SVN) -- linux, ended Fri Jun 26 20:39:27 2009 | 20:08 |
infobot | 3: 53d 8h 24m 4s running infobot 1.5.4 (SVN) -- linux, ended Tue Nov 7 05:34:20 2017 | 20:08 |
DocScrutinizer05 | ~+status | 20:08 |
infobot | Since Tue Sep 4 18:05:23 2018, there have been 0 modifications, 1 question, 0 dunnos, 0 morons and 1 command. I have been awake for 2m 59s this session, and currently reference 119767 factoids. I'm using about 80032 kB of memory. With 0 active forks. Process time user/system 1.59/0.15 child 0/0 | 20:08 |
DocScrutinizer05 | why did she ignoe your ~die ? | 20:09 |
DocScrutinizer05 | ignore* | 20:09 |
DocScrutinizer05 | ~+help die | 20:09 |
infobot | no help on die. Use 'help' without arguments. | 20:09 |
DocScrutinizer05 | hmmm | 20:09 |
TimRiker | can't see why ssl is not working. It connects, logs in, but then hangs. | 20:10 |
DocScrutinizer05 | I'd bet this once been a command overleyed by a factoid, for those lacking the permissions | 20:10 |
DocScrutinizer05 | yeah, refer to my ssh session lockup | 20:11 |
DocScrutinizer05 | seems to me like a OpenSSH compatibility problem | 20:11 |
DocScrutinizer05 | ~literal die | 20:13 |
infobot | "die" is "<action> takes two shots to the head and crumples to the ground, lifeless." | 20:13 |
DocScrutinizer05 | so much for the factoid. I'm not sure the command ever been listed in ~help | 20:14 |
DocScrutinizer05 | [Sun 06 November 2016] [22:27:36] <DocScrutinizer05> die | 20:19 |
DocScrutinizer05 | [Sun 06 November 2016] [22:27:41] Quit infobot (ibot@rikers.org) has left this server (Quit: cyal8r). | 20:19 |
DocScrutinizer05 | TimRiker: should I try ssh login reboot again? | 20:20 |
DocScrutinizer05 | maybe the sshd log on your end reveals something? | 20:21 |
TimRiker | ~chandel #wowace | 20:22 |
infobot | Prefix channel with '-' to delete for sure. | 20:22 |
TimRiker | ~chandel -#wowace | 20:22 |
infobot | Deleting channel #wowace for sure! | 20:22 |
infobot | Leaving #wowace... | 20:22 |
TimRiker | ~save | 20:23 |
infobot | saved user and chan files | 20:23 |
TimRiker | sure. go ahead and try it out. :) | 20:23 |
DocScrutinizer05 | reboot?` | 20:23 |
DocScrutinizer05 | ok | 20:23 |
DocScrutinizer05 | MHMMMM | 20:24 |
DocScrutinizer05 | TimRiker: https://pastebin.com/XFKZUuCj | 20:25 |
TimRiker | hmm. command not found. full pathing... | 20:26 |
TimRiker | try again? | 20:27 |
DocScrutinizer05 | at least it *works* now \o/ | 20:27 |
DocScrutinizer05 | sure | 20:27 |
DocScrutinizer05 | GREAT! | 20:27 |
DocScrutinizer05 | TimRiker: excellent diagnostics, just need to get rid of my own crap in between now, by removing the "-vvv" parameter to ssh, and keeping this diagnostics persistent ( ssh ... >>ibot-reboot.log) | 20:31 |
DocScrutinizer05 | aaaawww | 20:31 |
DocScrutinizer05 | :-) | 20:32 |
TimRiker | restarted to disable debug. | 20:35 |
TimRiker | ~chaninfo | 20:35 |
infobot | 118/118 channels, 3649 users, 2830 unique: #debian/1055, #kde/735, #devuan/251, #asterisk/171, #maemo/134, #gsoc/124, #wowuidev/97, #oe/91, #neo900/82, #utah/53, #arm-netbook/43, #harmattan/42, #asterisk-dev/40, #openmoko/40, #qi-hardware/36, #cubieboard/35, #bzflag/34, #maemo-ssu/33, #openmoko-cdevel/32, #elinux/26, #wowhead/25, #curseforge/22, #debianppc/20, #meego/15, #brlcad/15, #webos-internals/15, #uphpu/13, #bzflag-chat/13, ... | 20:35 |
TimRiker | removed from #wowace as they have ssl required, and alas, I can't get infobot working over ssl. | 20:36 |
DocScrutinizer05 | aaah, disable debug. Yeah, I was about to say: | 20:39 |
DocScrutinizer05 | HMMMM | 20:39 |
DocScrutinizer05 | 18:27.32 !DEBUG! main::IsNickInChan[1035] iNIC: hash channels contains mixed cased chan!!! | 20:39 |
DocScrutinizer05 | 18:27.32 LOG: last message repeated 233 times | 20:39 |
DocScrutinizer05 | TimRiker: users love it :-) http://maemo.cloud-7.de/irclogs/freenode/_devuan/latest.log.html#t2018-09-04T19:21:59 | 20:43 |
DocScrutinizer05 | now if only we could solve the mystery why the bot is doing a restart each night 00:18 UTC | 20:47 |
TimRiker | indeed. :) | 20:47 |
DocScrutinizer05 | next time I see her vanish, I might fire up a ssh reboot command and have a look at the diagnostics, maybe the ps gives away something useful | 20:48 |
* DocScrutinizer05 glares at logrotate | 20:48 | |
DocScrutinizer05 | and cron-daily | 20:48 |
DocScrutinizer05 | do the bot logs start at 00:21 UTC by any chance? | 20:51 |
DocScrutinizer05 | and previous rotate-saved one ends at 00:18? | 20:52 |
DocScrutinizer05 | I'd not feel too surprised if there was a ibot command "reopen-logfiles" and systemd-crond never heard of this command and instead restarts the bot like it does with every generic service on logrotate | 20:57 |
DocScrutinizer05 | loosely on topic: https://dev1galaxy.org/viewtopic.php?id=2345 | 20:58 |
DocScrutinizer05 | TimRiker: looking at diagnostics, I wonder if you do pstree *after* killall ibot | 21:06 |
TimRiker | ok | 21:07 |
DocScrutinizer05 | ok? | 21:08 |
TimRiker | changed. pstree before and after kill now. | 21:08 |
DocScrutinizer05 | is this what you do, and do you think it's ok? or do you swap the sequence? | 21:09 |
DocScrutinizer05 | aah :-) | 21:09 |
TimRiker | it should show the loop in a sleep 10 after the kill. might not have killed it yet though, might still be shutting down. | 21:09 |
DocScrutinizer05 | cool | 21:09 |
DocScrutinizer05 | ps before reboot for sure is more interesting than after, at least for diagnosing the reason why it had to get rebooted | 21:10 |
DocScrutinizer05 | also CPU percent, used network bandwidth up/down during last few seconds | 21:13 |
TimRiker | I added uptime there. thoughts on a good bandwidth command? | 21:15 |
TimRiker | remember the server has both ipv4 and ipv6 interfaces. | 21:15 |
DocScrutinizer05 | no idea, seems tricky though *should* be straight simple | 21:18 |
* DocScrutinizer05 browses through all /proc/<PID>/* | 21:18 | |
DocScrutinizer05 | TimRiker: [quote] NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. | 21:23 |
DocScrutinizer05 | https://pastebin.com/Zq8kaVBN nethogs --help | 21:25 |
DocScrutinizer05 | interactive though https://i.imgur.com/PUTbtKt.png -c 1 results in useless empty display | 21:28 |
DocScrutinizer05 | TimRiker: nethogs -t | 21:34 |
DocScrutinizer05 | TimRiker: nethogs -t -c 20 | 21:34 |
TimRiker | needs root, and I don't see a way to show only the current process. | 21:36 |
TimRiker | added a netstat -ntp|grep perl | 21:39 |
DocScrutinizer05 | yep, prolly both points are not solvable | 21:48 |
DocScrutinizer05 | the second maybe with a grep | 21:49 |
DocScrutinizer05 | looking at /proc/self/net/stat/nf_conntrack it's quite some gibberish ;-) | 21:51 |
DocScrutinizer05 | no idea if huimans can decode /proc/self/net/dev and if it has useful info | 21:52 |
DocScrutinizer05 | as long as your system stillprovides netstat :-/ I hate it when they phase out old reliable excellent tools in favor of new "do it all" monsters | 21:59 |
DocScrutinizer05 | https://dougvitale.wordpress.com/2011/12/21/deprecated-linux-networking-commands-and-their-replacements/ | 22:01 |
DocScrutinizer05 | >:-( | 22:01 |
DocScrutinizer05 | gotta love the "Replacement: Not apparent" | 22:07 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!