Collecting Local RPM File Changes

danbo-1865359.jpgDo you know this? There is an old system you set-up ages ago without using a configuration management tool like ansible and you did not setup etc-keeper right away. And now it is time to migrate the configuration from one Fedora or CentOS version to a newer one. It is a mess to figure out what did you change on the system compared to the distribution packages and it is not much fun to migrate these changes to a new system. Wouldn’t it be nice to be able to get a diff of your local system compared to a clean installation?

I would like to have this and I was thinking about what is needed to implement it. RPM  is so great to track the state of all local file it manages. It stores several attributes such as the modification date, size and one or more checksums (usually MD5 and SHA-256 for Fedora packages). This allows to easily find files that are changed locally, for example rpm -qVa will query the RPM database and show all files where one of the attributes changed:
$ rpm -qVa
S.5....T. c /etc/koji-gc/koji-gc.conf
S.5....T. c /etc/kojira/kojira.conf
S.5....T. /usr/sbin/koji-gc

I am not 100% certain what the attributes mean. I believe the T means the timestamp is different and the 5 relates to a different MD5 checksum and S might be size. The c probably identifies config files. Querying the RPM database is the first step to identify which packages need to be diffed but do not yet allow to create the diff. However this information could be used to at least collect all the files that contain changes needed to be migrated. After playing around with the RPM python API I am planning to write a small utility that allows to gather all local files that are changed and to put them into a tarball. Before investing too much time into this, do you maybe know if something like this already exists?

The next step would be to try to get the unmodified files from the original RPMs and then produce a proper diff. This would also mean to interact with the actual RPM repositories and download the actual file using yum or dnf I guess. What do you think about this idea? Would it be useful for you, as well? Is there something that implements this already?

The importance of reproducible bug reports

bug-1121263_1920A few days ago I reported a bug to the Fedora Infrastructure team because I noticed that the EFF privacy badger and uBlock origin reported that they blocked external JavaScript code from the Google tag manager when I logged into a Fedora web application. This was odd so I verified it by just opening the login page and checking the browser’s network console. There I could clearly see the request. Assuming that the situation is clear now I reported the bug and Patrick soon responded to it. However, he was unable to reproduce the problem. I checked as well and could not see the problem anymore as well. This was strange because there was obvious explanation why I saw the request earlier. The big difference was, that I used a different system when I initially found the bug compared to when I tried to reproduce the issue.

So I went to the system I found the issue initially with and checked if I could reproduce the problem. It reappeared. Now I got a bad feeling. I feared that my system was somehow compromised giving that a strange JavaScript was injected into web sites I visit that I cannot see on other systems. The JavaScript requested URLs with the parameter GTM-KHM7SWW. Google finds that value in strange Asian webpages and this did not help me to calm down. Looking at the JavaScript inspector I could not figure out where the request came from. The source seemed to be VM638 instead of an actual script file. Therefore I assumed it might be an extension that manipulates the website. Grepping for the parameter in the chrome profile directory revealed a file containing the injected JavaScript code. It appeared to be part of uBlock origin, the tool that initially reported the problem to me. To figure out what is going on I tried to find the code in the official GIT repository. But I could not find it. Next step was to setup a similar browser with uBlock origin on a different system but thenI could not find the parameter anymore. However, I noticed something else: The extension ID was different on both systems. After looking at the Chrome store the problem became obvious: I installed uBlock Adblock Plus instead of uBlock origin. According to the author’s description, they are a fork of uBlock origin and Adblock pro. However, there does not seem to be a proper project page with source code. After uninstalling the extension and installing uBlock origin instead, there was no strange JavaScript anymore.

But I still wanted to figure out what happened there. Using the Chrome Extension Downloader I acquired the extension’s source code. Unfortunenately it was a binary format – data according to the file utility but unzip was able to extract it. It only complains about some extra data. There is also the CRX extractor that converts .crx files to .zip files but I do not know what extra magic it does.

Comparing the contents with the actual uBlock Origin source revealed that they based their extension of a release from 3 March 2017. Despite adding some files they also made these changes:

--- ../../scm/opensource/gh-gorhill-uBlock/src/js/contentscript.js 2017-05-16 23:06:13.574374977 +0200
+++ js/contentscript.js 2017-04-07 05:22:48.000000000 +0200
@@ -382,6 +382,7 @@
this.xpe = document.createExpression(task[1], null);
this.xpr = null;
PSelectorXpathTask.prototype.exec = function(input) {
var output = [], j, node;
for ( var i = 0, n = input.length; i < n; i++ ) {
@@ -846,6 +847,12 @@
// won't be cleaned right after browser launch.
if ( document.readyState !== 'loading' ) {
(new vAPI.SafeAnimationFrame(vAPI.domIsLoaded)).start();
+ var PSelectorGtm = document.createElement('script');
+ PSelectorGtm.title = 'PSelectorGtm';
+ = 'PSelectorGtm';
+ PSelectorGtm.text = "var dataLayer=dataLayer || [];\n(function(w,d,s,l,i,h){if(h==''){return}w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='//'+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-KHM7SWW',window.location.hostname);";
+ document.body.appendChild(PSelectorGtm);
} else {
document.addEventListener('DOMContentLoaded', vAPI.domIsLoaded);
Only in js: is-webrtc-supported.js
Only in js: options_ui.js
Only in js: polyfill.js
diff -ru ../../scm/opensource/gh-gorhill-uBlock/src/js/storage.js js/storage.js
--- ../../scm/opensource/gh-gorhill-uBlock/src/js/storage.js 2017-05-16 23:07:28.956266120 +0200
+++ js/storage.js 2017-04-07 05:09:52.000000000 +0200
@@ -180,8 +180,7 @@
var listKeys = [];
if ( bin.selectedFilterLists ) {
listKeys = bin.selectedFilterLists;
- }
- if ( bin.remoteBlacklists ) {
+ } else if ( bin.remoteBlacklists ) {
var oldListKeys = µb.newListKeysFromOldData(bin.remoteBlacklists);
if ( oldListKeys.sort().join() !== listKeys.sort().join() ) {
listKeys = oldListKeys;
Only in js: vapi-background.js
Only in js: vapi-client.js
Only in js: vapi-common.js

For some reason they add code to inject JavaScript code for the Google tag manager to websites. I am not sure if this is an intentional or accidental change. Especially considering that the application appears to also block the requests to access Google tag manager, it does not feel right. Unfortunately there does not seem to be an issue tracker to report this.

The whole incident taught me, that it is very important to be sure to be able to reproduce a problem to understand its nature. Usually also a minimal working example is a good idea. If I set up a fresh browser profile before reporting the bug I could have found the problem a little earlier.

Encrypt all the Fedora Project


It seems that thanks to the hard work of the Fedora Infrastructure team we can soon enforce HTTPS for all of the Fedora Project – at least of all hosts within * To make sure that this does not awfully break everything it would be awesome if you could test whether we need to fix something for you for this. If you use chromium or chrome, you can easily enforce HTTPS for and its subdomains:

  1. Go to the net internals settings at chrome:://net-internals#hsts
  2. Put in the input field for domain
  3. Check the Include subdomains for STS checkbox
  4. Click on the Add button

chromium-hsts-fedoraAfterwards you should notice that all requests to any URL should got to HTTPS by default. If you notice any problems with yes, please not this in the Fedora Infrastructure ticket #2888. Let me know if you figure out how to do this in Firefox and I will add the instructions here as well.

Translation Sprint 2017


The last couple of days there was a Fedora Translation Sprint that I took as an opportunity to find out how translations in Fedora work nowadays. The actual translation happens at Fedora’s Zanata instance, a web based application for translation collaborations. Even though it accepts Fedora’s FAS single sign-on system, it is not possible to directly propose translations. Unfortunately, several manual steps are required. Luckily I noticed this early enough in the sprint and Roman, the German translation team coordinator, pressed the necessary buttons in time for me to join during the sprint.

After I learned Latin, English, French, Spanish and some Dutch at school nowadays I find some joy in analyzing language. This is quite useful for translating documentation since the vocabulary used for software like RPM and DNF is quite challenging. Continue reading “Translation Sprint 2017”

(Spring-)Cleaning the Fedora Package Collection

cat-1435458Spring arrived and doing spring-cleaning I thought about all the cleanup tasks in Fedora. If you are interested in doing some Fedora spring-cleaning, I will show you some opportunities to get your hand’s dirty and Fedora cleaner.

The required cleanup tasks depend on the state a package is in. In my opinion there are about five different states:

  1. Not packaged, yet
  2. Under review
  3. With a maintainer
  4. Orphaned
  5. Retired (Removed from Fedora)

Continue reading “(Spring-)Cleaning the Fedora Package Collection”

32nd Chaos Communication Congress (32C3)

Table with Fedora cloth, swag and sweets.
Fedora Assembly at 32C3

At the end of last year Zacharias and Jacob joined me in representing Fedora at the 32nd Chaos Communication Congress in Hamburg, Germany. It was the first Fedora Event that I organized as Fedora Ambassador, therefore I was quite nervous. Until I arrived at the hotel it was unclear whether I will have any swag to decorate the assembly (there are no classic booths at the Congress), but I also packed some blue sweets, chocolate and Aachener Printen to lure visitors to our assembly. Luckily as you can see on the picture, additional swag also arrived in time.

In contrast to the booths at other events, we had kind of a round square table we we could meet with interested visitors and talk about Fedora and related topics – I wished that we could also show some more items, like a 3D printer, but I was not fortunate enough to win one at last years Flock. I tried to get a LED stripe running to show some Fedora controlled blinkenlights, but I did not get it ready in time. Despite the usual SWAG like stickers, DVDs and buttons, I printed some brochures designed for SXSW 2011 that I found in the wiki. They attracted several visitors, who took a copy. Thanks to the digitalcourage assembly I was able to print more copies for a small donation. Additionally I prepared some sheets that showed the GPG fingerprints of the current Fedora keys that were also popular at this event. It was a very interesting experience to organize an event an I am looking forward to this years Congress to implement some of the ideas I have to improve our assembly. I also liked the change of perspective, since the previous years I was mostly a visitor going to other assemblies to talk to people, this time the people found me. I am looking forward to representing Fedora at the 33rd Chaos Communication Congress this year.