Path of Exile Wiki:PyPoE: Difference between revisions

From Path of Exile Wiki
Jump to navigation Jump to search
>OmegaK2
No edit summary
 
(40 intermediate revisions by 9 users not shown)
Line 1: Line 1:
[https://github.com/OmegaK2/PyPoE PyPoE] is a python 3 based collection of development tools to work with Path of Exile developed by [[User:OmegaK2]].  
'''PyPoE''' is a Python 3 based collection of development tools to work with Path of Exile, originally developed by [https://github.com/OmegaK2 OmegaK2]. The command line interface is used extensively to update many pages on the wiki with game data.


It is currently in a [[{{FULLPAGENAME}}#Development|development alpha]], which means it is neither feature complete nor well-tested and subject to frequent changes.
The [https://github.com/Project-Path-of-Exile-Wiki/PyPoE Project Path of Exile Wiki fork of PyPoE] is on github. Follow the [https://github.com/Project-Path-of-Exile-Wiki/PyPoE/wiki/PyPoE-101:-Installation-and-setup instructions] to install it.


__TOC__
__TOC__


==Overview==
==Usage on the wiki==


===Goals of the Project===
PyPoE is used on the wiki to help with exporting game data to the wiki to reduce the manual work load from editors.


* provide a mostly complete and stable python3 api to work with Path of Exile and it's files
There are generally no restrictions placed on who can use PyPoE, but please note that without bot or admin access regular users are subjects to some limits in regards to automated usage. It is possible however to simply write something to your local disk (using the --write option) and then manually apply it to the wiki.
* have a stable and useful UI binding to browse the files
* provide CLI tools for extracting files for convince
* provide utilities to publish the data on the Path of Exile wiki


===Features===
The process of uploading images is generally manual; for PyPoE to be able to export uncompressed images see the [[Path_of_Exile_Wiki:Community_portal#Uncompressed_DDS_files_from_the_ggpk_for_the_wiki|instructions on the community portal]].
* Development Library
* Graphical User Interface (UI or GUI)
* Command Line Interface (CLI)


==Installation==
After major game updates updates to the script may be required for it to work correctly. Issues can be reported on GitHub or on [[User:OmegaK2]] if they are wiki-related.


These installation instructions are currently aimed at development builds.
==Wiki exporter parameters==


===Prerequisites===
These are parameters that are supported on most commands.


====Python 3.4====
Some parameters may not be available depending on the command, and some commands have have many additional parameters available. Please consult the command line help <code>-h</code> for details on specific parameter.


Windows:
{| class="wikitable"
* Navigate to https://www.python.org/downloads/ and download the latest version of python 3.4
! Parameter
* Install and follow the instructions on screen
! Arguments
* In particular make sure the Python executable as well as the scripts folder are located in the %PATH% variable
! Description
|-
| -h
| {{n/a}}
| Shows the help message which includes a list of all possible parameters.
|-
| -p
| {{n/a}}
| Prints the output of the command to the console.
|-
| -wr
| {{n/a}}
| Writes the output of the command to the folder specified in the config.
File names will be named similarily to the wikipages, usually prefixed by the type of export (e.x. [[Imbued Wand]] would be called <code>item_Imbued Wand.txt</code>)
|-
| -d
| file path
| Overrides the config output directory and outputs to this one instead.
|-
| -w
| {{n/a}}
| Enables automatic exporting to the wiki.
This requires a complete installation of the CLI in particular including mwclient to access the wiki API.
|-
| -w-u
| username
| Name of the user account that will be used to authenticate when editing the wiki via PyPoE.
|-
| -w-p
| password
| Password of the user account that will be used to authenticate when editing the wiki via PyPoE.
Please note the password will be visible in the console. [[Special:BotPasswords|Bot passwords]] may be used instead of your account's main login credentials, although it is not required.
|-
| -w-dr
| {{n/a}}
| Peform a dry run without actually editing any pages. This is useful to check if manual changes on a page would be accounted for correctly or overriden.
|-
| -w-oe
| {{n/a}}
| Only edit existing pages and do not create any new ones.
|-
| -w-slp
| time in seconds
| Sleep inbetween editing request.
The wiki may rate limit you, if the default settings do not work for you, consider using a high value here.
|-
| -w-mt
| number of threads
| How many simultaneous threads to run. Avoid using.
|-
| --format
| &lt;template&#124;module&gt;
| This can be used on some command to export a module string instead of a template. This can be used to write to a file to get module strings for module debugging purposes.
|-
| -im
| {{n/a}}
| Allows the extraction of related images on some commands
|-
| -im-c
| {{n/a}}
| Converts the extracted image from DDS to PNG format. This requires [https://github.com/zao/poe-utils/releases process-image] to be available from the command line.
|}


Linux:
==Wiki export commands==
* Ubuntu/Debian: apt-get install python3


====Git====
===Items===


Windows:
{| class="wikitable"
* Navigate to https://git-scm.com/ and download the latest version of git
! Command
* Make sure git is added to the %PATH% variable on windows
! Description
! English<br>Wiki<br>only
|-
| <code>pypoe_exporter wiki items item</code>
| Base items (normal items)
| {{no}}
|-
| <code>pypoe_exporter wiki items prophecy</code>
| Prophecies
| {{no}}
|-
| <code>pypoe_exporter wiki items maps</code>
| For the 5th map series (Betrayal and newer). For older series base item export is used.
DOES NOT include unique maps
| {{no}}
|-
| <code>pypoe_exporter wiki items atlas_icons</code>
| Special icons for the interactive atlas.
| {{yes}}
|}


Linux:
===Pages in data name spaces (monsters, mods, areas, passives)===
* Ubuntu/Debian: apt-get install pip


====QT4 (UI only, Linux only)====
{| class="wikitable"
! Command
! Description
|-
| <code>pypoe_exporter wiki area</code>
| Exports the area data
|-
| <code>pypoe_exporter wiki mods</code>
| Exports the modifier data
|-
| <code>pypoe_exporter wiki monster</code>
| Exports monster data (experimental)
|-
| <code>pypoe_exporter wiki passive</code>
| Exports passive skill data
|-
| <code>pypoe_exporter wiki skill</code>
| Exports skill data


See http://pyside.readthedocs.org/en/latest/building/linux.html for details.
Please note exports of non-player skills are unsupported and will result in errors. This export only needs to be run for mods related skills, not skill gems (skill gems will populate their skill data automatically).
|}


===Generic data dumps for lua modules (quest/vendor rewards, atlas, blight, delve, etc) ===
Due to the nature of using lua and cargo on mediawiki, these particular changes will not be loaded without null-editing. Please see the corresponding templates for details on which template to null-edit for changes to go through.


===Installing PyPoE===
Generally the data is stored in <code>Template:Name/cargo/table_name/N</code> where <code>Name</code> is the name of Template usually after whatever it is doing, <code>table_name</code> is the name of the database table, and <code>N</code> is one for each every 1000 entries.


It is highly recommended to use the development build if no stable release is available, so updates from git apply to the install.
Please note that some module pages might require admin privileges to edit to prevent vandalism.  


Linux:
* On many distributions, python and pip can link to either version 2 or version 3 depending on the system defaults, instead consider using
** <code>python3</code> instead of <code>python</code>
** <code>pip3</code> instead of <code>pip</code>
====Navigating to the folder====
* Open a command line (Windows: cmd.exe )
* Navigate to the folder you want to have PyPoE installed to/from
* Run <code>git clone https://github.com/OmegaK2/PyPoE/tree/dev/</code>
* Go into the checkout folder where <code>setup.py</code> resides
====Running pip====
To use the development install (it makes the files editable), specify <code>-e</code>.
By default, PyPoE only installs the dependencies for the core (PyPoE.poe). However, there are additional packages available:
{| class="wikitable"
{| class="wikitable"
! Group
! Command
! description
! Description
!
|-
|-
| ui
| <code>pypoe_exporter wiki lua atlas</code>
| GUI support
| Exports data for the current atlas of worlds layout
| [[Template:Atlas]]
|-
|-
| cli
| <code>pypoe_exporter wiki lua bestiary</code>
| CLI support
| Exports data related to bestiary crafting
| [[Template:Bestiary]]
|-
|-
| cli-wikibot
| <code>pypoe_exporter wiki lua crafting_bench</code>
| Support for the --wiki option in the wiki exporter
| Exports crafting bench data such as recipes and unlocks
| [[Template:Crafting bench]]
|-
|-
| dev
| <code>pypoe_exporter wiki lua delve</code>
| Development libraries (i.e. for tests & generating documentation)
| Exports delve data such as depth scaling, sulfite costs and delving upgrades
| [[Template:Delve]]
|-
| <code>pypoe_exporter wiki lua monster</code>
| Exports shared monster data such as the level progression, scaling for magic/rare monsters, map scaling info, monster types and resistances data, etc.
| [[Template:Monster]]
|-
| <code>pypoe_exporter wiki lua pantheon</code>
| Exports pantheon data
| [[Template:Pantheon]]
|-
| <code>pypoe_exporter wiki lua ot</code>
| Exports base statistics for monsters and player characters
| [[Template:Generic stats]]
|}
|}


The base command for installing is:
==Support in other language wikis==
 
PyPoE can work in other language wikis since it uses the appropriate game files for other languages, however there a number of original work and custom strings that need to be translated. This needs to be supported directly in the code, I'd recommend setting pull-requests or translations for the relevant portions so they can be added.
<code>pip install -e .</code>
 
If you wish to add any extra packges, add them in a comma-separated list in brackets behind the dot:
 
<code>pip install -e .[ui,cli,cli-wikibot,dev]</code>
 
==Reporting an issue==
 
First, make sure the issue isn't known already:
* https://github.com/OmegaK2/PyPoE/issues
* check the individual files TODOs


If not, post an issue on github and provide the following
Currently Russian (provided by [[User:Ruba159753‎]]) and German is available.
* title: short description of the bug
* detailed description of the bug
* if available, provide the relevant traceback
* if available, provide the relevant code that caused the bug  (i.e. if your own code)


You can also submit pull requests that to help fix bugs, I'll review (and possibly) edit them. You agree they'll placed under the MIT license. Also see [[{{FULLPAGENAME}}#Contributing|Contributing]]
The following files need strings for the specified variables:
* PyPoE/_data/custom_descriptions.txt
* PyPoE/cli/wiki/parser.py
** _inter_wiki_map
** _HIDDEN_FORMAT
* PyPoE/cli/wiki/parsers/lua.py
** _ITEM_MAP
** _TWO_STONE_MAP
* PyPoE/cli/wiki/parsers/item.py
** _conflict_resolver_prophecy_map
** _NAME_OVERRIDE_BY_ID
** _LANG
* PyPoE/cli/wiki/parsers/area.py
** _LANG
* PyPoE/cli/wiki/parsers/incursion.py
** _incursion_room_page_name


==Contributing==
==External links==
* [https://github.com/Project-Path-of-Exile-Wiki/PyPoE Project Path of Exile Wiki fork of PyPoE]
* [http://omegak2.net/poe/PyPoE/ OmegaK2's documentation for PyPoE]


Everyone is generally welcome to contribute to the project.
[[Category:Development information]]
 
[[ru:Path of Exile Wiki:PyPoE]]
There are basically 3 ways to do so:
* Feature requests & reporting issues
* Sending pull requests
* Becoming a dev
 
===Pull requests===
 
You're welcome to send a pull request on github for features or changes you think that should be in PyPoE.
 
Please keep a few things in mind:
* the code in the request should be well behaved and a proper fix or addition
* the change should be under the MIT license
* if it's an entirely new feature, it may debatable whether it is has a place in PyPoE
* the submitted code should:
** be well behaved and a proper fix or addition
** be [https://www.python.org/dev/peps/pep-0008/ PEP8] compatible (minus the line-length)
** include changes in the respective tests or new tests
** validate against existing tests (if tests were changed, validate against those)
* backwards incompatible changes are more suited for the dev branch
 
So for example:
 
* Likely to be accepted
** general improvements to existing code of UI, CLI or API
** new, well-behaved features that extend the existing functional
** support for missing file formats
** updated dat.specification.ini
** additional tests
* Likely to be rejected
** changes unrelated to the goal of the project
** refusal of making the change itself available under MIT license
** changes with poor coding style
 
===Become a dev===
 
If you want to become an '''active''' developer and meet the requirements please contact me. I'll manually unlock people for access to the repo.
 
Notes:
* If you just want to contribute a few changes, there is no need to become a dev and you can send pull requests instead.
* If you just want your own repo, just fork the project on github
 
Requirements:
* You should
** have a good amount of experience with developing in Python 3
** be willing to actively contribute, i.e. making changes on your own, working on the TODOs
** be fluent in English (written); no you don't have to be a perfect speaker, but you're English should be good enough to have no issues with communication
** be available in the IRC channel
* I'll want to see some pieces of code you've written; a good history of pull requests will suffice, otherwise:
** another open source project you've been involved in,
** some private things you've written but are willing to let me have a look at
 
It would be extra helpful, if you:
* have experience with C/C++ and writing embedded python libraries with C/API
* have a lot of experience with Path of Exile [for the api]
* are experienced with using Mediawiki [for exporter]
* speak other languages fluently or natively [for translating the project]
* are adept with reverse engineering [to help with api]
 
==Structure==
 
{| class="wikitable"
! Folder
! Description
|-
| /
| The root folder of the project
|-
| /PyPoE/
| The root python folder
|-
| /PyPoE/_data/
| Shared data used by the other files. For example, the specifications for the .dat files reside there.
|-
| /PyPoE/poe
| The core API to work with Path of Exile; this will be of primary interest to other developers that want to use the API.
|-
| /PyPoE/shared
| Core libraries and functions that are not specific to path of exile
|-
| /PyPoE/cli
| Libraries and functions for the command line interface of PyPoE
|-
| /PyPoE/ui
| Libraries and functions for the QT4 based user interface of PyPoE
|-
| /scripts/
| Collection of scripts intended to be invoked from the command line.
|-
| /tests/
| pytest based tests for the library and scripts
|}

Latest revision as of 04:31, 16 October 2021

PyPoE is a Python 3 based collection of development tools to work with Path of Exile, originally developed by OmegaK2. The command line interface is used extensively to update many pages on the wiki with game data.

The Project Path of Exile Wiki fork of PyPoE is on github. Follow the instructions to install it.

Usage on the wiki

PyPoE is used on the wiki to help with exporting game data to the wiki to reduce the manual work load from editors.

There are generally no restrictions placed on who can use PyPoE, but please note that without bot or admin access regular users are subjects to some limits in regards to automated usage. It is possible however to simply write something to your local disk (using the --write option) and then manually apply it to the wiki.

The process of uploading images is generally manual; for PyPoE to be able to export uncompressed images see the instructions on the community portal.

After major game updates updates to the script may be required for it to work correctly. Issues can be reported on GitHub or on User:OmegaK2 if they are wiki-related.

Wiki exporter parameters

These are parameters that are supported on most commands.

Some parameters may not be available depending on the command, and some commands have have many additional parameters available. Please consult the command line help -h for details on specific parameter.

Parameter Arguments Description
-h N/A Shows the help message which includes a list of all possible parameters.
-p N/A Prints the output of the command to the console.
-wr N/A Writes the output of the command to the folder specified in the config.

File names will be named similarily to the wikipages, usually prefixed by the type of export (e.x. Imbued Wand would be called item_Imbued Wand.txt)

-d file path Overrides the config output directory and outputs to this one instead.
-w N/A Enables automatic exporting to the wiki.

This requires a complete installation of the CLI in particular including mwclient to access the wiki API.

-w-u username Name of the user account that will be used to authenticate when editing the wiki via PyPoE.
-w-p password Password of the user account that will be used to authenticate when editing the wiki via PyPoE.

Please note the password will be visible in the console. Bot passwords may be used instead of your account's main login credentials, although it is not required.

-w-dr N/A Peform a dry run without actually editing any pages. This is useful to check if manual changes on a page would be accounted for correctly or overriden.
-w-oe N/A Only edit existing pages and do not create any new ones.
-w-slp time in seconds Sleep inbetween editing request.

The wiki may rate limit you, if the default settings do not work for you, consider using a high value here.

-w-mt number of threads How many simultaneous threads to run. Avoid using.
--format <template|module> This can be used on some command to export a module string instead of a template. This can be used to write to a file to get module strings for module debugging purposes.
-im N/A Allows the extraction of related images on some commands
-im-c N/A Converts the extracted image from DDS to PNG format. This requires process-image to be available from the command line.

Wiki export commands

Items

Command Description English
Wiki
only
pypoe_exporter wiki items item Base items (normal items)
✗
pypoe_exporter wiki items prophecy Prophecies
✗
pypoe_exporter wiki items maps For the 5th map series (Betrayal and newer). For older series base item export is used.

DOES NOT include unique maps

✗
pypoe_exporter wiki items atlas_icons Special icons for the interactive atlas.
✓

Pages in data name spaces (monsters, mods, areas, passives)

Command Description
pypoe_exporter wiki area Exports the area data
pypoe_exporter wiki mods Exports the modifier data
pypoe_exporter wiki monster Exports monster data (experimental)
pypoe_exporter wiki passive Exports passive skill data
pypoe_exporter wiki skill Exports skill data

Please note exports of non-player skills are unsupported and will result in errors. This export only needs to be run for mods related skills, not skill gems (skill gems will populate their skill data automatically).

Generic data dumps for lua modules (quest/vendor rewards, atlas, blight, delve, etc)

Due to the nature of using lua and cargo on mediawiki, these particular changes will not be loaded without null-editing. Please see the corresponding templates for details on which template to null-edit for changes to go through.

Generally the data is stored in Template:Name/cargo/table_name/N where Name is the name of Template usually after whatever it is doing, table_name is the name of the database table, and N is one for each every 1000 entries.

Please note that some module pages might require admin privileges to edit to prevent vandalism.

Command Description
pypoe_exporter wiki lua atlas Exports data for the current atlas of worlds layout Template:Atlas
pypoe_exporter wiki lua bestiary Exports data related to bestiary crafting Template:Bestiary
pypoe_exporter wiki lua crafting_bench Exports crafting bench data such as recipes and unlocks Template:Crafting bench
pypoe_exporter wiki lua delve Exports delve data such as depth scaling, sulfite costs and delving upgrades Template:Delve
pypoe_exporter wiki lua monster Exports shared monster data such as the level progression, scaling for magic/rare monsters, map scaling info, monster types and resistances data, etc. Template:Monster
pypoe_exporter wiki lua pantheon Exports pantheon data Template:Pantheon
pypoe_exporter wiki lua ot Exports base statistics for monsters and player characters Template:Generic stats

Support in other language wikis

PyPoE can work in other language wikis since it uses the appropriate game files for other languages, however there a number of original work and custom strings that need to be translated. This needs to be supported directly in the code, I'd recommend setting pull-requests or translations for the relevant portions so they can be added.

Currently Russian (provided by User:Ruba159753‎) and German is available.

The following files need strings for the specified variables:

  • PyPoE/_data/custom_descriptions.txt
  • PyPoE/cli/wiki/parser.py
    • _inter_wiki_map
    • _HIDDEN_FORMAT
  • PyPoE/cli/wiki/parsers/lua.py
    • _ITEM_MAP
    • _TWO_STONE_MAP
  • PyPoE/cli/wiki/parsers/item.py
    • _conflict_resolver_prophecy_map
    • _NAME_OVERRIDE_BY_ID
    • _LANG
  • PyPoE/cli/wiki/parsers/area.py
    • _LANG
  • PyPoE/cli/wiki/parsers/incursion.py
    • _incursion_room_page_name

External links

ru:Path of Exile Wiki:PyPoE

Retrieved from "https://www.poewiki.net/index.php?title=Path_of_Exile_Wiki:PyPoE&oldid=1099460"