devtools: add a script to fetch and postprocess translations

Run this script from the root of the repository to update all translations from transifex.
It will do the following automatically:

- create a transifex configuration file
- fetch all translations
- post-process them into valid and committable format
This commit is contained in:
Wladimir J. van der Laan 2014-04-30 09:46:04 +02:00
parent 58c01a376d
commit 9dd5d79244
3 changed files with 82 additions and 3 deletions

View file

@ -69,3 +69,15 @@ If there are 'unsupported' symbols, the return value will be 1 a list like this
.../64/test_bitcoin: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
.../64/test_bitcoin: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
update-translations.py
=======================
Run this script from the root of the repository to update all translations from transifex.
It will do the following automatically:
- fetch all translations
- post-process them into valid and committable format
- add missing translations to the build system (TODO)
See doc/translation-process.md for more information.

View file

@ -0,0 +1,66 @@
#!/usr/bin/python
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT/X11 software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Run this script from the root of the repository to update all translations from
transifex.
It will do the following automatically:
- fetch all translations using the tx tool
- post-process them into valid and committable format
- remove invalid control characters
- remove location tags (makes diffs less noisy)
TODO:
- auto-add new translations to the build system according to the translation process
- remove 'unfinished' translation items
'''
from __future__ import division, print_function
import subprocess
import re
import sys
import os
# Name of transifex tool
TX = 'tx'
# Name of source language file
SOURCE_LANG = 'bitcoin_en.ts'
# Directory with locale files
LOCALE_DIR = 'src/qt/locale'
def check_at_repository_root():
if not os.path.exists('.git'):
print('No .git directory found')
print('Execute this script at the root of the repository', file=sys.stderr)
exit(1)
def fetch_all_translations():
if subprocess.call([TX, 'pull', '-f']):
print('Error while fetching translations', file=sys.stderr)
exit(1)
def postprocess_translations():
print('Postprocessing...')
for filename in os.listdir(LOCALE_DIR):
# process only language files, and do not process source language
if not filename.endswith('.ts') or filename == SOURCE_LANG:
continue
filepath = os.path.join(LOCALE_DIR, filename)
with open(filepath, 'rb') as f:
data = f.read()
# remove non-allowed control characters
data = re.sub('[\x00-\x09\x0b\x0c\x0e-\x1f]', '', data)
data = data.split('\n')
# strip locations from non-origin translation
# location tags are used to guide translators, they are not necessary for compilation
# TODO: actually process XML instead of relying on Transifex's one-tag-per-line output format
data = [line for line in data if not '<location' in line]
with open(filepath, 'wb') as f:
f.write('\n'.join(data))
if __name__ == '__main__':
check_at_repository_root()
fetch_all_translations()
postprocess_translations()

View file

@ -71,14 +71,15 @@ We are using https://transifex.com as a frontend for translating the client.
https://www.transifex.com/projects/p/bitcoin/resource/tx/
The "Transifex client" (see: http://support.transifex.com/customer/portal/topics/440187-transifex-client/articles)
will help with fetching new translations from Transifex. The Transifex configuration (`.tx/config`)
is used to fetch new translations from Transifex. The configuration for this client (`.tx/config`)
is part of the repository.
It is also possible to directly download new translations one by one from the Transifex website.
Do not directly download translations one by one from the Transifex website, as we do a few
postprocessing steps before committing the translations.
### Fetching new translations
1. `tx pull -a`
1. `python contrib/devtools/update-translations.py`
2. update `src/qt/bitcoin.qrc` manually or via
`ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale\/\1.qm<\/file>/'`
3. update `src/qt/Makefile.am` manually or via