Benutzer:DrTrigonBot/ToDo-Liste/DONE

aus Wikipedia, der freien Enzyklopädie
Vista-gaim.svg
Hinweis: Bitte diese Seite NICHT editieren! (Please DO NOT edit this page!)
Stop hand nuvola grey.svg
Ja, wirklich. Es ist schön, wenn jemand vorbeikommt und Fehler oder Links korrigiert, aber bitte nicht hier!

Files related to DrTrigonBot-"framework" (look also at Botwiki:Python:DrTrigonBot scripts):

Configuration: config.py, user-config.py, (others?)

Data: (BoW training, ...)

Look also at Benutzer:DrTrigonBot#Source.

Aktuelle Version

Aktuelle Revision (FishEye): Activity • Files • Users • RSS feed
Id What Script
Bugfix for: framework, dtbext_query.py, runbotrun.py
25 Spezial:Permanentlink/64368286 sandbox for new created status/subster bot. This bot watches text on external (or internal) pages, the text is specified by regex. If this text changes, the bot copies the new/actual text into the wiki. Tag model to mark places of text drop is borrowed from MerlBot (and this sub-bot could also be used on my local wiki with an anacron-job ;). subster.py
Version 0.2.0000

Version 0.2.0000

Revision 1 (WebSVN): Log
Id What Script
Feature/Bugfix/... for:
  • Experimentally introduced a new bot part 'page_disc.py' for Spezial:Permanentlink/77229254#Disk-Zusammenfassung_auf_WD:ADT, test is running on Benutzer:DrTrigon/Spielwiese#Test 3. For this several small changes had to be made on 'dtbext_query.py' and 'dtbext_wikipedia.py' which should also return some benefit to other bot parts, e.g. 'sum_disc.py' (parser and others - look at the comments in the mentioned files).
  • MailerBot; the bugfixes in 'dtbext_query.py' enabled to send longer mails and thus it looks like the earlier introduced long mail support becomes obsolete. May be for very special needs it can still be helpful, but at the moment it is disabled.
 page_disc.py, dtbext_query.py, dtbext_wikipedia.py, mailer.py
Feature for:
  • MailerBot; enabled the bot to create and sent long mail through the toolserver for users, that have registered themselves.
 mailer.py, mailerreg.py
Feature for:
  • SubsterBot; enabled the bot to retrieve wiki text content, not only html page content. Used for wiki to wiki text substitutions.
 subster_beta.py, substersim.py
Bugfix for:
  • Small bug in MailerBot; the 'content_maxlen' has to include the subject size and thus must be calculated for every page (else some mails for certain pages cannot be sent).
 mailer.py
Bugfix for:
  • Small bug in 'Pages.getVersionHistory(...)'; if the API omitts some entries (and sometimes it does) and does not return at least a 'missing' for them, the class does recognize this now and generate and add dummy results for the missing pages.
 dtbext_wikipedia.py
Bugfix for: framework, dtbext_query.py, runbotrun.py
Bugfix for:
  • Spezial:Permanentlink/72481024#Dein Bot: komische Einträge Notification links (section matching); strange behaviour. When using wiki API to check sections, both methods are now used always, instead of only when the first returned nothing and the longer output is used (since the other might not be complete). This should prevent problems with pages like Spezial:Permanentlink/72550929 with any kind of include-tags. This method now is slower since always 2 calls are made, this should be improved. (bot run now takes about 1.5h instead of about 1h before; 50% slower...)
 dtbext_wikipedia.py
B50 Spezial:Permanentlink/64115999 Notification links; don't work. Used target link from http://toolserver.org/~merl/UserPages/query.php?user=DrTrigon&format=xml (parameter/value 'url'). But the handling of foreign-wiki-links in framework could still be improved (interwikimap). dtbext_pagegenerators.py, sum_disc.py
37 Creation of a stable release v0.2.0000 based on bug fixes, new feature and improvements in current semi-stable release v0.1.0013. Testing of this version and update of bot software on toolserver to this new version. Extensive testing of all new functions and whole code like output of times (correct timezone setting) and others.

Setup of new CRON-job for compression of history once all 14 days.

(toolserver)
MailerBot created. This sub-bot is very experimental at the moment; it does it's job for me, probably/hopefully for you to, but do not expect miracles at this current state (but ideas, hints and bug reports are still very welcome). mailer.py
36 Check CPU load of whole bot script with: 
/usr/bin/time -v python /home/drtrigon/pywikipedia/runbotrun.py -cron

because the bot runs every day at least for 1 hour, so the load must not be too high! If it is, try to delay the expensive calls and/or contact the toolserver admins what to do, because 'nice' is already in use. The load of history compression has not to be checked, since the script runs not that often.

Result: 

drtrigon@nightshade:~$ /usr/bin/time -v python /home/drtrigon/pywikipedia/runbotrun.py -cron
	Command being timed: "python /home/drtrigon/pywikipedia/runbotrun.py -cron"
	User time (seconds): 32.23
	System time (seconds): 2.86
	Percent of CPU this job got: 1%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 46:18.79
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 0
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 42316
	Voluntary context switches: 204822
	Involuntary context switches: 3707
	Swaps: 0
	File system inputs: 0
	File system outputs: 1816
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

should be ok with 1% CPU load.

(toolserver)
B44 For several pages related with MerlBot the bot was not able to retrieve section (heading) info. Those issues appeared on following pages:

and were related to <noinclude> and <onlyinclude> tags, and have been solved. But the pages are handeled like templates because of 'action=parse&text={{%s}}__TOC__&prop=sections' is used for recognition. Therefore it could become necessary in future to use a more sophisticated way; get page content, preprocess (handle <noinclude>, <onlyinclude>) it, pass it to 'action=parse&text=%s__TOC__&prop=sections' and then postprocess (handle the rest like <pre> and others) the result.

dtbext_wikipedia.p
F45 List of parameters/options enhanced:
  • checkedit_list: each User has the ability to add additional patterns for pages that should be monitored by the bot.

(similar to Benutzer:CopperBot/Überwachte Seiten)

Mentioned by Benutzer:Merlissimo (was already implemented in bot code).

(publishing)
B49 Bugfix for:

Spezial:Permanentlink/63464336#Problem_mit_DrTrigon_Bot has solved itself, according to Spezial:Permanentlink/63713171.

34 Poll/survey under the bot users about happiness and feature requests. (my talk page)
F38 New feature integrated, by using data from http://toolserver.org/~merl/UserPages/query.php?user=DrTrigon&format=xml the bot is now able to give users hints if any of their other wiki talk pages was changed, just for the case of getting messages there.

Mentioned by Benutzer:Merlissimo in chat.

dtbext_wikipedia.py, dtbext_pagegenerators.py, sum_disc.py, sum_disc_conf.py
B47 Bugfix for:

Reported by [[ Forrester ]].

sum_disc.py
F46 As long as F45 is not implemented yet, please add

to the surveillance list.

Mentioned by Benutzer:Merlissimo.

sum_disc_conf.py
B48 Bugfix for:

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look at id B45

Reported by [[ Forrester ]].

(look there)
B46 Bugfix for:
  • Spezial:Permanentlink/62158091 / 25. entry; the brackets '[' and ']' have to be replaced in some parts of the output (namely the url part of the links).
 sum_disc.py
B45 Bugfix for:
  • Spezial:Permanentlink/62087384 / 2nd last entry; is reported despite the fact it is signed/not necessary to report. The problem was that a user used english page links e.g. 'User:...' in its signature, those describers were added now.
 sum_disc_conf.py
From id 44 splitted into B?? and F?? for simpler counting.
33 Bot runs all 24 hours (instead of all 48 hours).

Proposed by Benutzer:Forrester (for BLUbot).

toolserver
List of paramters/options enhanced:
  • altsign_list: especially for BLUbot introducted parameter to give alternative user names to check for signature, e.g. you have a representative or something else.
 sum_disc.py, sum_disc_conf.py
40.2 Bugfix for:
  • Spezial:Permanentlink/61457254 / last block, 1st entry; the last part of '_checkThData()', check for text after signature recognizes the '|}' as additional text, code to prevent this was tested but not activated beacuse the better way would be to change the welcome template and suppress additional newlines at the end.
 sum_disc.py
42, 43 Bugfix for:

The bot reports redirects (created by users) as new discussions (which of course are non-existent). Maybe redirects should be resolved and the resulting pages checked against the ignore-list? This change was mentioned by Benzen C6H6 09:20, 5. Jul. 2009 (CEST) (probably a hamster?).

dtbext_wikipedia.py, dtbext_query.py, sum_disc.py
17.2
  • Wanted to change the history format such that each time, at each bot run the last history file entry is read but contains the full actual history. This data is updated and checked for changings since last run, the actual version history for each page is retrieved and saved - this would serve a simple programming interface, however it uses way to lot of hard drive space (and memory). Now using the old incremental format, but with the data updated and checked for changings since last run and 'compressHistory' does no additional relevancy check, just rewrites same history content (bot behaviour keeps the same after compression). So fast compression was removed completely.
Dogma: 'compressHistory()' must not do any kind of relevancy/signature check, and not alter the bots behaviour.
  • At bot run (not compression) all discussions are checked for signature and if it is completely missing the discussion for this page is closed and it is removed from history.
  • Additionally 'rollbackHistory()' was implmented to make the bot to use older history entries only.
  • Different pagestates introduced to make the new history handling practible:
    • _PS_warning: serious or no classified warnings/errors that should be reported
    • _PS_changed: changed page (if closed, will be removed)
    • _PS_unchanged: unchanged page
    • _PS_new: new page
    • _PS_closed: closed page (remove it from history)
    • _PS_maintmsg: maintenance message
  • Split of 'self._news' into 'self._news' (_PS_changed, _PS_new, _PS_maintmsg) and 'self._others' (_PS_unchanged, _PS_warning) and furthermore parallel (to 'self._news') updating of 'self._history' (_PS_changed, _PS_new, _PS_maintmsg, _PS_unchanged, OHNE _PS_closed) which is saved in the end to the history file.
  • History should be dropped at least all two week, because it is bigger now (could be zip-ed)!
 sum_disc.py
35.2 Method of choice for wiki content access is to use the API and retrieve pages in bunches (of e.g 10 pages). Dumps are very fast too but not available on a daily base (all 2 days only), look at http://download.wikimedia.org/dewiki/ also (Benutzer:MerlBot claims to use dumps). So switched to handling of page read access with API (it looked to me that the recent rev. of bot-framework have switched partly to access over API, but that is wrong; it are a few commands only, mostly user or config related), this was already implemented with different methods in dtbext.wikipedia.Page().get(). Now it was implemented for simultaneous read of a bunch of pages with prop=revisions (rv) in dtbext.wikipedia.Pages().get(). But this has a small drawback, the parameter/option 'directgetfull_switch' becomes a kind of useless or obsolete (because it is faster to re-read single pages per block, then splitting the block and all handling related with this), therefore removed parameter/option:
  • directgetfull_switch

It is important that the original and the new page read methods deliver the same page content!

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 17

dtbext_wikipedia.py, dtbext_query.py, sum_disc.py
21 Merged with most recent pywikipedia-framework 6977 and testing of new functions, as well as update of wikipedia.py (because of lines 725-728). The bot reports now on every run the most recent framework rev. because of verbosity and information (download via svn could be automated, look also at Benutzer Diskussion:Benzen#Fragen wegen Bot-Modifikation). The extensions needed for the bot are now techn. adapted to the framework and packed into one single package with same structure like the framework:
  • wikipediaAPI dtbext.wikipedia and dtbext.pagegenerators (analogue to the framework)
  • wikipediaAPI.ContributionsGen Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} dtbext.pagegenerators.UserContributionsGenerator (is now a generator and returns a list of pages instead of a list of tupels)
  • wikipediaAPI.BacklinksGen Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} dtbext.pagegenerators.ReferringPageGenerator (as above), furthermore dtbext.wikipedia.Page().getReferences() with API implemented. If the API is used for call is not related to the generator but to the given page-typ. The new generator supports setting of limit/number additionally.
  • wikipediaAPI.GlobalWikiNotificationsGen Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} dtbext.pagegenerators.GlobalWikiNotificationsGenerator (experimental only)
  • wikipediaAPI.Page()._getParsedString() dtbext.wikipedia.getParsedString()
  • wikipediaAPI.Page().getFull(), wikipediaAPI.Page().getParsedContent() and wikipediaAPI.Page().get() (the same as wikipedia.Page().get()) dtbext.wikipedia.Page().get() with mode='full' , mode='parse' or without any setting (default)
  • wikipedia.sectionencode for bot messages could be possible to solve the old special char issue
  • dtbext.wikipedia.html2notagunicode introduced (as addition to 'Unicode library functions')
  • clean_sandbox2.py Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} clean_user_sandbox.py
  • dtbext.query introduced
 (various)
35.1 Introduced bundeling of API calls. A new class 'Pages' performs all actions on a set of pages simultaneously. Currently page version history information can be retrieved this way, this can be enhanced on other functions too.

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 17, 35.2

wikipediaAPI.py, sum_disc.py
Full compression (not fast) improved by adding a part that removes all old/removed headings from history page entries. wikipediaAPI.py, sum_disc.py
  • New history format (v5) introduced; use dict (again) is the one and only intelligent/correct way to hold unique data
  • Conversion of old history formats in '_getHistoryPYF' during read-in, instead of different places spread all over the code. This causes all history data to be keept in up-to-date format, now.
Dogma: history format for headings (with checksums) has to be a dict, with unique keys. (if not possible to make unique, then add a timestamp or something like this)
sum_disc.py
Heavily improved parameter/option handling, all internal used user specific bot parameters/options can be modified by the users. The behaviour either to overwrite or preserve the defaults can easily be configured internally. sum_disc.py, sum_disc_conf.py
18 List of paramters/options enhanced:
  • ignorepage_list: here you can add pages to ignore, to this list added are your own discussion pages. if you do not define this parameter your archiv pages are used by default.
 sum_disc.py, sum_disc_conf.py
17.1 Speed-up by:
  • pre-compile of all often used and static regex (probably do some more later, if it makes sense)
Dogma: frequently used regex must be pre-compiled.
wikipediaAPI.py, sum_disc.py
28 Clean the code by removing all unnecessary 'try' constructs, replace '(... == None)' contructs by the shorter (and cleaner) version and add documentation.

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 26, 27

sum_disc.py
32 Compression could be done by comparing against the backlinks. This was implemented and tested, here the observations: the original method depends on '_SearchForSignature(...)', the alternative method on a properly configured 'backlinks_list'. if properly configured the alternative mode is a lot faster, but it depends on this config, whereas the original method works always! the statement all pages listed on backlinks have currently a signature is correct if 'backlinks_list' is properly configured (how to check this?) sum_disc.py
27 The checksum checker in '_checkThData(...)' improved (no need for 'try' structure anymore) and the different modes documented. Mode v1 was removed because it is very old, if there is any history entry with v1, then it is ignored, the page is reported and logged again. This should cause not that much problems (since there are very few or no such entries existent anymore). sum_disc.py
31 Checkout of API functions/calls, here some examples:
  • meta=userinfo (ui) with 'hasmsg': gives only a hint if there is a new message or not (already implemented in bot framework) but how to set it read after loading the page to make the message disappear is still left unclear / function not needed at the moment
  • meta=allmessages (am): can be used to translate some describtions and other things from English to this wiki / function useful
  • meta=siteinfo (si): a lot of site, wiki and server info / function useful
  • action=paraminfo: obtain information about certain API parameters / function useful

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 13, 24

24 The page Spezial:Änderungen an verlinkten Seiten (Special:RecentChangesLinked) could be implemented as an addition to Spezial:Beiträge (Special:Contributions) here is an example for DrTrigon. Unfortunately I was not able to find an API for this, but list=backlinks (bl) which is the same as Spezial:Linkliste (Special:WhatLinksHere) and this was implemented as an addition. The use of this is configurable by a new parameter/option:
  • getbacklinks_switch: read the backlinks to user discussion page and check them additionally
 wikipediaAPI.py, sum_disc.py
29 All pages 'Wikipedia:Löschkandidaten/<DATE>' are read twice (second time with 'getFull()'), therefore a parameter/option was introduced:
  • directgetfull_switch: read all pages with 'Wikipedia:Löschkandidaten/\d' at first attempt with 'getFull()'

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 17

(This option may later become obsolete, e.g. when updating to new bot framework; if the default read method 'get()' is changed)

sum_disc.py
26 Bot should run without any use of 'fall back' mode in '_getThContent(...)' therefore this obsolete, old and unreliable mode was removed but replaced with a warning message (just in case). sum_disc.py
Version 0.1.0013

Version 0.1.0013

Id What Script
List of paramters/options enhanced:
  • altsign_list: here you can add other signatures to check/handle equivalent like the users signature. Needed for Benutzer:BLUbot and WP:DÜP service.
 sum_disc.py
40.1 Bugfix for:

All those are related with the different history formats and the way they were read in and updated. All older formats are converted now during read and in to v5. Took from Version 0.1.0014.

sum_disc.py
Bugfix for: Spezial:Permanentlink/61457165 / 1st entry; head recognition out of sync, added another check for validity (to decide to use full read mode). Took from Version 0.1.0014. wikipediaAPI.py
Bugfix for: html tag heading recognition; tags with additional values are now recognized too. wikipediaAPI.py
Bugfix for: heading recognition; e.g. Spezial:Permanentlink/61381636 / last block, 1st entry; because of some <nowiki>-tags the function had some problems. All '=' within those tags are substituted by '&#61;' as work-a-round. wikipediaAPI.py
Bugfix for: re-reported discussions; e.g. Spezial:Permanentlink/61311960 / last entry or Spezial:Permanentlink/61309424 / 1st entry of last block; this is related with the different history formats and the way they were read in and updated. Some older formats are converted now during read and in that way the data can be updated properly. sum_disc.py
Bugfix for: twice appearing headings; the API returns wrong links they are not complete, the numbering (from 2 upwards) is missing. So a fix for this was implemented now the bot should really be able to get all head links correct. wikipediaAPI.py
The DrTrigonBot status panel was added to the Toolserver Wiki as a tool (with description page, a.o.) and a bit modified (added a link to User:DrTrigon). panel.py
35.3 Bugfix for: Spezial:Permanentlink/61105710 / 1st entry of last block; the page was processed without any heading info, in fact it could not retrieve the headings (sections). Added an additional/alternative method if the first fails.

look also at id 28

wikipediaAPI.py
35.4 Bugfix for: Spezial:Permanentlink/61106764 / 2nd entry; the headings got out of sync, in the 2nd half it was shifted about 1. The problem was that headings recognized by wiki parser has to start at the first char of a line!

Fehler beim Parsen (MathML mit SVG- oder PNG-Rückgriff (empfohlen für moderne Browser und Barrierefreiheitswerkzeuge): Ungültige Antwort („Math extension cannot connect to Restbase.“) von Server „https://wikimedia.org/api/rest_v1/“:): {\displaystyle \rightarrow} look also at id 28

wikipediaAPI.py
5 testing of new 0.1.0013 release. upload of it and rearrange data on toolserver.
  • copy 'local:pywikipedia-devel_0.1.0013/' to 'toolserver:./pywikipedia/'
  • copy 'toolserver:./pywikipedia/' to 'toolserver:./public_html/source/pywikipedia/'
  • copy 'toolserver:./pywikipedia/' to 'local:toolserver/pywikipedia/' and 'local:toolserver/public_html/source/pywikipedia/'
 local, toolserver
15 it would be a good idea to use https://jira.toolserver.org/secure/BrowseProjects.jspa the Toolserver issue tracker if the amount of bugs reported and to handle, exceed a certain number, but for the moment it is not needed. the problem is you cannot add a new project as normal user, so the system here has to deliver this service. BUGS
16 had issues with creation of BOT-MESSAGE containing some special chars (e.g. ':'). the algorithm should now be more stable, but it estimates the correct heading only, so could still make some problems.
Dogma: use as simple bot maintenance message texts as possible.
sum_disc.py
13 sample bot output: 
* Processing User List (wishes): Benutzer:DrTrigonBot/Diene Mir!
NOTE: You have new messages on wikipedia:de
...

if such a message apears again, the bot now read its discussion page Benutzer Diskussion:DrTrigonBot and prints/show the content to output (log file). that informs you about the message and should lead to supress further display (of the same old message).

runbotrun.py
14 a very intressting fact is the page Spezial:Änderungen an verlinkten Seiten (Special:RecentChangesLinked) could be able to replace or be an alternative to Spezial:Beiträge (Special:Contributions). here an example for DrTrigon (linked to Benutzer Diskussion:DrTrigon) to show the capabilities. sum_disc.py
9 if the bot is executed by CRON job on toolserver the preprocessing/filtering of log file output does not work properly. have added another regex to the output handler, hope that helps. runbotrun.py
10.3 purge page cache through wiki API according to example http://toolserver.org/~drtrigon/wiki-purge.html (has to be deleted from the toolserver now) implemented as wikipediaAPI.Page(...).purgePageCache(). the issues occured here earlier have disappeared or were due miss-programming/-understanding of the API by myself. wikipediaAPI.py, sum_disc.py
10.2 the render given in http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikiparser/ looks to be (very) old; the newest contributions are Modified Mon Oct 29 06:41:05 2007 UTC (19 months, 1 week ago) and the wiki API seems to do a quite good job. so the bot is using the API at the moment. wikipediaAPI.py, sum_disc.py
11 because of improvements in 10.1 before the bot is now able to detect level 1 headings (through a work-a-round, but it is possible) with the old it was nearly impossible. so level 1 headings are taken into account except the bot has to fall-back into the old mode so the wrong display of discussions in front of level 1 headings (on problematic pages such as Löschdiskussionen bei Benuterseiten, WP:FzW, WP:Auskunft, Portale and others) should stop now. wikipediaAPI.py, sum_disc.py
10.1
  • wikipediaAPI.Page(...).getSections() improved; now it is on a good level and able to find and synchronize the subsection headings in pages
  • wikipediaAPI.Page(...).getFull() introduced; to help ...getSections() it uses 'expandtemplates' API and therefore should be able to give a low level output of wikitext on pages, however it seems to hide HTML comments for example...
  • _checkRelevancyTh(...) improved with help of ...getSections() but it still has full fall-back capabilities.
  • wikipediaAPI.Page(...)._getParsedString(...) introduced and wikipediaAPI.Page(...).getParsedContent() improved; they should work now.
  • _parseNews improved because of the two mentioned before it was possible to replace _renderWiki(...) (and with it _renderWikiTempl(...) too) with _getParsedString(...) and they are useless now and can be deleted.
 wikipediaAPI.py, sum_disc.py
7 '_setUser(...)' and '_getUsers()' used both two seperate parameter/option handling mechanisms. since the entry in Benutzer:DrTrigonBot/Diene Mir! without and parameters in brackets '{...}' offers already a config option; you can specify any target page within your userspace. this option and the "real" options in brackets where handeled in 'self.userListInfo' and 'self.param'. those two mechanisms were now merged and the target page in 'self.userListInfo' is added to 'self.param' as 'userResultPage'.
Dogma: 'self.param' is the only option/parameter handling mechanism (especaly for params changable by all users through wiki)
sum_disc.py
6 config management/system for running bot in seperate file; all options (global vars) moved there. the variables could be named nicer (more self-explainig) but the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!
Dogma: the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!
sum_disc.py, sum_disc-conf.py
8 extended search/check range/context:

added. some of them could be extended by Wikipedia:Administratoren/* (but this would trigger more traffic with few benefits only). this change was mentioned by Benutzer:Benzen in Benutzer Diskussion:DrTrigon/Archiv#Bot: Reklamation.

sum_disc.py
Version 0.1.0012

Version 0.1.0012

Id What Script
18 parameters/options introduced with configuration directly on Benutzer:DrTrigonBot/Diene_Mir!

currently possible/implemented options are:

  • checkedit_count: how many recent edits should be checked (mainly for use with BLUbot)
  • reportchanged_switch: should changed (and already as new reported) discussions on change be listed again (mainly for use with BLUbot)
 sum_disc.py
12 format/system of saved checksums had to be changed once again (3rd or 4th time) because of the dict used with subheadings as key was not unique (all subheadings on a page may have the same name); now the bot is using a list again, but a list of tuples (there are still some issues open with subheadings, this may make another change necessary...) sum_disc.py
4 add interface for history's compression; was done initially but does not make sense since the permissions of user 'apache' will not allow to to re-write the history. the good possibilities are to do it by hand (manual) with a toolserver login (from time to time) or setup a e.g. monthly or weekly CRON job
Dogma: history compression should be done by hand or CRON job (if not by hand, do auto backup)
panel.py
3 improvement of admin log interface: 'filelist' and 'checkfiles' are not necessary, the whole path can be generated only once just before the call of 'os.remove'! panel.py
2 instead of using 'runbotrun.py -auto', now a CRON job was created on toolserver, this should in order take care on the bot timing and be able to deal with server reboots. with this action all issues related with this are solved. call is of bot is 'runbotrun.py -cron' runbotrun.py
1 admin interface added for removal of old log files instead of using cyclic logs (5 files e.g.)
Dogma: admin interface is used for log file removal - it's link is not published
panel.py
Abgerufen von „https://de.wikiup.org/index.php?title=Benutzer:DrTrigonBot/ToDo-Liste/DONE&oldid=223092503“