There are a couple of cases that are worth separating out.

In case we are able to automate away programming entirely, this worry is ill-founded. I would liken it to escalator operators–while we still enjoy the services the provide today, there are no more elevator operators, and no one knows how to operate one shy of pressing a button. That’s okay.

What happens when, as seems likely for at least the foreseeable future, we end up in a hybrid world, where machines and engineers continue to work together to create code? In that case, it seems that a lot of the easy, low-hanging fruits can be largely automated away, and what remains for the human is to steer and surgically intervene for the most difficult code pieces. But juniors are ill-positioned to do this.

In the software industry, new grads typically have to prove themselves by undergoing an almost torment-like regime as code monkeys, pushing out as much code as possible. You can most easily observe this in competitive internships, which are organizationally setup to select for the interns with the highest amount of throughput. There are arguments to be made for this selection strategy: it gives juniors a large amount of experience quickly, it instills an objective (even if ridiculous) criterion, and, lastly, it largely seems to just work. Without a doubt, it also filters for motivated people, who want to do a lot. Strangely, I have also noticed that at this level quantity and quality are often positively correlated. However, in this new AI age, where code is a commodity, producing large amounts of code will no longer be a differentiation criterion. This has two implications:

1. The need for junior engineers (where the bulk of grunt work code is written) significantly declines.
2. Companies will need to look for a new metric on which to rate junior engineers – thanks to AI, anybody could produce huge amounts of code. In particular, I worry for the differential ability of the best junior engineers to stick out the most.

While worrying about this long term would be foolish–ostensibly, if there was a continued need for programmers, then people would also find a way to fill it–one might argue that we see the effects of this already, and it is this intermediate generation that is affected the hardest: I heard from a couple of my University professor friends that Computer Science applications are decreasing for the first time in decades (beware, this is very early data from a handful of personal connections for one or two semesters that might well be random), as are job postings for junior engineers across the industry.

I have another worry from a more education-specific angle: If AI can get the code right 99% of the time, how do you ever learn, or motivate to learn, the leg work of getting type definitions right, of figuring out that regular expression, or how to fix this nasty runtime error? Now, you might say, that a) the history of computing is one of abstraction and AI is simply one step up and that b) we have had tools such as calculators, and yet, people still know the mechanics of multiplication, if they had to do it. I agree, but would also say that AI is different: calculators are largely not stochastic machines, at least not for things such as multiplications. AI, however, is–there is always the chance that its answer might be wrong–hallucinated or otherwise. So it provides a leaky abstraction at best. Safe for bugs in the compiler, we can safely forget about machine code. We cannot, however, do this in the AI age: We are simply not used to the compiler working only 99% of the time. From a practical perspective today, as long as there is a real chance for unintended behavior, we need to maintain the knowledge and skills to dive into them and fix them. I’m not exactly sure how to do this, but it seems obvious that we need to strengthen the entire debugging, testing, and verification arm in our Computer Science curricula.

Obviously, Software Engineering or coding will not be the only area affected by AI. But as it stands, it is one of the most heavily influenced, and the one where changes are deployed the fastest. It is a fascinating time to be part of the AI wave from the inside of where it all happens, but I am glad I can do it from a slightly elevated spot. Little time before the rising tide that may give.

In general, my concerns with Android Auto can be categorized into two themes:

1. Poor Reliability (PR): Stuff works 1 out of 3 times, then fails in ways that render the system useless. I know that (sufficient) network isn’t always available, and so things can’t always work, but the same is true for mobiles in general and they do have reasonable ways around it. It seems to me as if Android Auto assumes there is always connectivity, and if it isn’t, then it just can’t recover.
2. Missing (or Regressed) Features

Android Auto could be so much better if … it were like Apple Car Play?

The list of complaints

• Saturday, May 20th 2023: Let’s start with the good news: I got the new flexible UI layout (https://developers.google.com/cars/design/android-auto/design-system/layout for adaptive-responsive framework) and can now accept or decline incoming WhatsApp calls, even using the buttons on my steering wheel! Nice! What still does not work is initiating a WhatsApp call. I also still have this really weird bug that upon starting the voice assitant, there is a break in the audio, then for 2-3s the current song plays really loudly, then gets toned down. After this, I can start to speak, which usually works. Upon switching to a different song, the current default seems to be ‘silence is good’ (i.e., there is no confirmation, nothing), but I have to switch to Spotify to activate playing. Which is a tad distracting, but at least it was working reliably yesterday.
• Saturday, Dec. 31st 2022: After declining a call on the mobile (for whatever reason, I still cannot accept or decline incoming WhatsApp calls in Android Auto, but have to do it on the phone) twice, my phone wouldn’t connect to the car anymore. At all. I had to stop the car, lock it, wait outside for a minute and try again before I could get it to connect to Android Auto again.
• On June, 24th, 2022, Android Auto while navigating on Google Maps, just completely froze and did not update anymore. When I re-connected, the car did launch Android Auto but the screen stayed black. Only rebooting the mobile helped. This was very frustrating because I was in an area with many turns unknown to me.
• Since 2022: Whenever I am switching over songs in Spotify via Google Assistant, I would expect the song to start playing automatically (particularly, if I was listening to Spotify before). This is the behavior it shows on Apple Car Play, and it works always there. In Android Auto, before sometime in 2022, you had to manually navigate to the Play button because after switching songs, the default state of the player would be paused. Annoying (and a small fix, one might think), but at least that was reliable. Since medio-2022, the song auto-plays, but almost always without sending any sound. So the icon on Spotify shows ‘playing’, but no sound comes out. To get it to work, click the pause button, wait for some time, pray 3 times to the Greek nymph Echo, who can only sing, not talk, and hit the play button again. If you are lucky, the song begins. If not, it might begin in a couple of seconds. Or not at all. Repeat until you are either gone crazy, hit an obstacle in front of your car, or do get some sound out.
• Since 2021: Whenever I activate Google voice assistant (e.g., via the steering wheel button on my Mercedes), the background sound should get silenced so the mic can record what I am saying. Instead, what happens is that the music gets elevated for the first three seconds of the mic being active, then suddenly stops. Super strange behavior. I assume what they intended for is for the sound to fade out over the three seconds, but this never worked.

Der ADAC und andere kommen in Praxis-Tests auf leicht widersprüchliche Ergebnisse: mal wird ein theoretischer Mehrverbrauch von 3% festgestellt, mal gar eine Spriteinsparung. Wie sieht es also wirklich theoretisch aus, und welche Praxis-Regel kann man ableiten, ab welchem Preisunterschied es sich lohnt E10 statt E5 zu tanken?

Grundsätzlich sind Super E10 und Super E5 Benzin bis auf den Anteil an Biosprit Ethanol identisch: E10 muss zwischen 5 und 10% Bioethanol enthalten, E5 maximal 5%. Bioethanol unterscheidet sich von mineralischem Öl in zwei wichtigen Eigenschaften:

• Die Klopffestigkeit von Bioethanol ist höher
• Seine Energiedichte ist geringer (6,28 kWh/l für Bioethanol zu 8,9 kWh/l für Superbenzin)

Wenn wir nun von einer durchschnittlichen Differenz in der Bioethanolkomponente von 5% zwischen E5 und E10 ausgehen, so erhalten wir pro Liter eine durchschnittlich um 0,131kWh geringere Energiedichte für E10, oder 1,49%, im Vergleich zu E5. Nun hat ein moderner Verbrennungsmotor noch nicht mal eine Effizienz von 50%, schafft also maximal 0,75% dieser Energie auch zu nutzen. Zusätzlich kommt noch hinzu, dass E10 aufgrund der höheren Klopffestigkeit möglicherweise effizienter verbrannt werden kann.

Somit folgt als Handregel: Sobald Super E10 also mindestens 1% günstiger als E5 ist, lohnt es sich in jedem Fall, E10 zu tanken. Die verlorene Reichweite durch die geringere Energiedichte wird durch einen günstigeren Preis über-kompensiert.

Stellen sich noch folgende andere Fragen:

• Verträgt mein Auto überhaupt Super E10? Unbedingt vorher checken, aber die meisten Autos ab Baujahr 2010 tun dies, jedenfalls antriebssseitig.
• Andererseits gibt es frühe Forschungsergebnisse vom Fraunhofer-Institut, die besagt, dass E10 den Aktivkohlefilter schädigen könnte
• Ist E10 kilmafreundlicher als E5? In der europäischen Union, wahrscheinlich schon. In anderen Ländern mag dies anders aussehen.

What is puzzling is that this Herculean task of reviewing in its current form hinges entirely on volunteers providing free reviews In Software Engineering, we have seen many recent help calls that say the current review system no longer scales, with some individuals providing more than 100 reviews annually. This problem is exaggerated by a system that has no memory of prior reviews across venues and by authors who re-submit articles with little to no changes to different venues, triggering a new set of reviews, essentially treating reviews as a free resource. Moreover, typically, the same set of people are found in Program Committees again and again. We also know that well-knonwn members of certain, already under-represented groups (such as women in Software Engineering) often take on a higher-than-average reviewing load, being invited over and over (perhaps to fill the gender gap?), while it can be be hard for others from the same minority group to be invited at all. This leaves the question of how one ensures that everyone in the community can and does contribute, if not equally, then in proportion to the number of articles they submitted. Some academics have hard-and-fast rules they commit to, but the range of how many reviews to perform among just three persons is huge, with a 4x difference. Clearly, we need a system to not be reliant on gut feeling or on individuals’ diverging gut feel for what constitutes a fair amount.

Why a simple system is bound to fail

It is immediately obvious that a simple system to ensure review load fairness does not work because papers are submitted by a (variable) number of authors, instead of just one. Moreover, the most simple idea to ensure fair reviews would otherwise be “You can submit here if you have reviewed here before” does also not work because (leaving aside the chicken-egg problem for a moment) because of a limited review pool for each venue and not all reviewers being able or qualified to review for a given venue.

A credit-based review system

I have long touted the idea of a credit-based review system, but never written about it publicly. In this post, I want to lay out the possible logistics of it. Similar to Machine Learning-based solutions, there might be some hyper parameter tuning and optimization in order, depending on how it plays out in practice. Regardless, the idea hinges on a substantial number of venues in a field taking part, which is an admittedly difficult to overcome initial hurdle. The system could be orthogonal to EasyChair or HotCrp, and connected to them, identifying submitting authors by their ORCID. Identification, a problem many sites face, should not be a hurdle, either, because authors have a very strong inherent motivation to submit under their (and only their) real name.

Submitting

Being allowed to submit to a venue requires a certain number of review credits being present. This could either be the sum among the authorship or taken from each author individually. Upon submission, X credits are deducted.

For beginning PhD students, or just to get the system started, every person signing up should be awarded a number of fixed credits (similar to when the Deutsch Mark was introduced in 1948, where every citizen received 40 Deutsch Marks). If we want to go fancy, we could make this dependent on the current position of the person: Perhaps an industry researcher is entitled to 2 credits, while a PhD student might receive 5.

Reviewing

Similarly, after delivering a review, a reviewer gets Y credits awarded. This system could tackle the problem of declining review quality, too: guarantee a minimum rate for a review, and depending on the quality, more credits could be given.

A similar thing could happen for more prestigious venues (actually, this would be perhaps the first objective marker for them to tout themselves as such) or journal manuscripts, where more credits could be granted.

Toward an economy of reviewing

Many academics currently decline invitations to PCs or, more severely, I think, journal paper reviews: I have come across a TSE article that had an unbelievable 40 academics rejecting to review; needles to say, I declined, too (I did find the person to finally review, it, though; thanks, Annibale!), for there is no tangible benefit of reviewing for them. As soon as you can not submit a paper because you do not have the required credits, this changes! The famous number 3 ICSE rule from 2017 could perhaps be avoided by this system, too: authors simply would not have been able to amass the required review credits.

When deciding on a pool of reviewers, I as a chair was recently faced with a candidate pool twice the required size of absolutely equally qualified candidates. In the end, decided to select among them by chance. A system could instead have looked at the number of reviews they each already performed in the past year.

A market-based system, I like the possible inherent fairness this would involve. Think about it to the extreme, there could even be a monetary price for a review credit (if, similar, to the European emission system, you want to allow transfer of credits): this could enable authors from industry who would not have an opportunity to review, to submit, too by buying credits.

Is reviewing really without benefits?

In this discussion, I have not really mentioned publishers much. This is because I think in a future academic system, there is simply no room for them. Alas they also would be the best place to kick-start this system, with the exception of IEEE/ACM, which would be even better (think about Elsevier requiring you to have credits for publishing there). Of course, you currently get the very questionable offer of a 50% discount on “the publisher’s” latest overpriced offerings.

Currently, much of the debate focuses on reviewing as solely an altruistic contribution to a functioning ecosystem. While I think that this is largely a valid stance, it falls somewhat short of reality: Receiving direct pay for reviewing is seldom, but does happen, like, for example, the comfortable rate reviewers for the European Union receive, for example for ERC grants (granted, I have applied and many people I know, too, however, no one ever heard back from them … strange). Apart from direct monetary benefits, reviewing (and, if you will, as an extension of reviewing) chairing gives one extra ordinary visibility within the community. This ‘academic’ fame can come in handy at numerous occasions, for example, establishing oneself as a senior academic or when externally certifying expert status, for example, for visa purposes.

Conclusion

In the end, the implementation of this system technically should be pretty straight-forward, and in my opinion, has benefits over simply spreading the gigantic review load of 27 full time positions in Software Engineering alone, fairly across a large number of people: It might help visibility of underrepresented groups, it would free up a large amount of time of the currently pretty elite reviewers, and it certainly would eliminate the free riders of the current system. Of course, many current reviewers would argue that they do this for free, because they like it, or because they are uniquely qualified. But in light of a systemic problem, I would love to have that discussion.

Of course, one could say that exceptional service is one way to get yourself known and contribute to the field. If we systemize that, this might go away. I guess this depends on the final market economics of reviewing, which I do not claim to forsee now. We can always tweak the system such that one does not get away with not doing any community work, while no else suffers. Another argument is that not every person is equally qualified to review, and I agree, too. However, assigning different number of credits per review and moving towards reviewing standards should alleviate these concerns.

Before I accepted the offer to chair, I asked myself three questions:

• Am I a good fit (litmus’s test: do I know the conference? Have I submitted there before?)?
• Who is my co-chair (super important, as it later turned out!)
• Do I bring something new to the table?

For the last point, I studied the previous years’ CfP. I noticed that it was somewhat in-cohesive (looked like several additions were made over many years, but no one really went over it in its entirety), innovations in reviewing were not present, and that it was a bit unclear what Industry track authors and reviewers really wanted. Therefore, we completely re-wrote the CfP.

Governing principles

When Miltos and I took over, we quickly agreed on three governing principles:

• Make the track as open and transparent as possible.
• Integrate innovations in reviewing.
• Reduce unnecessary workload on authors and reviewers.

Author wishes

Most SE conferences’ Industry tracks were still single-blind (and certainly of the premier conferences; with the exception of ICSME. A frequent counter argument against this was that important industrial context would often be lost, or make the submitted manuscript awkward to read. As a result, we made blinding optional for authors. I am happy to report that not only did 33% of submissions follow up on this choice, but also that there were no complications reviewing these blinded submissions.

Another important factor for authors is that anecdotally, many feel obliged/are forced to write rebuttals, even for fruitless submissions. To ensure this does not happen, we transparently communicated the paper’s score together with the initial set of two reviews; papers with a very negative assessment (at least one strong reject and another reject) would be auto-rejected, unless reviewers saw the need not to do so. For the 17 submission we early-rejected, no reviewer saw the need for this. Instead, it allowed authors to get early feedback, allowed them to re-work their submissions, and us to save the potential burden of a third review. Similarly, we early-accepted 14 papers, again saving the burden of (feeling to) having to write a review for submissions that would anyway make it in.

Reviewing wishes

In times of corona, it was really hard for us to predict how many submissions we would receive. In addition, unfortunately, we could not get that info for the 2020 edition of the Industry track. As a result, we predicted conservatively to receive 30% more submissions than 2019 (in fact, we received +50%!).

In addition, we wanted to give particularly more junior and Industry people a chance to review, and not limit it to our personal networks. As such, we designed a PC consisting 50% of people we know we could trust and 50% nominations, shared via Twitter (if anyone knows of a better platform to reach academics in SE, please let me know). We were unsure whether we should allow self-nominations (you do want to have reliable reviewers), and fortunately ended up doing so: 100% (!) of nominations were self-nominations, and self-nominated revierwers ended up not being less reliable than our contacts. However, we vetted all nominations. Because everyone seemed like a great fit and we had 50% more nominations than we needed, we simply randomly chose among the nominees.

What helped keep review load down was not only enlarging the PC, but in particular early accepts and rejects. Moreover, we think having 2 reviews for these submissions did not significantly influence review quality.

What we could improve

Even so, there was one main item (and two smaller things) that we would change if given the chance to chair again. Above, you see the review procrastination graph. What was a bit shocking to me was that after our internal deadline, 30% of reviews were missing! Personal reminders helped a lot initially, but then the influx of new reviews dwindled. Similar to stocks, by spreading the review load wide, we effectively hedged against a total default, but would have had to factor in a number of reviews being late. In all this, it was good that we left enough buffer between the internal deadlines and the officially announced ones. Still, the obvious solution for this would be to have an emergency PC of roughly 5-10% the size of the PC.

Moreover, perhaps we shared too much information with authors (making reviews immediately, even before we summarized them with an email), which caused a bit of confusion. Finally, there were a few submissions on which both Miltos and I were conflicted, so we had to override the conflicts.

What went well

We received a (likely) record number of 61 submissions, and quite accurately predicted this. Out of those, we accepted 23 (39%). With an increased PC and a new reviewing mode, we could still reduce the average review workload to 3.5 in the first round and only 1 paper in the second round. We were also able to acquire new reviewing blood via the (self-)nomination form. FSE was the first major SE conference to use optional double-blind submissions; we are happy to report that not only did a third of the submissions make use of this, but that it also caused no problems during reviewing. When we desinged all this in the beginning, we had no idea that HotCrp technically supported all our non-standard ideas (e.g., sending out early-rejections). While it definitely lacks a user friendly interface, we were at least able to botch everything. It was also great to work with Miltos, who was pro-active and helped out when I wasn’t available. Finally and most important, we were able to keep all deadlines.

Going fully virtual

While there was originally (and as recently as two months ago) hope that ESEC/FSE 2021 would be at least a hybird conference (with a pyhsical event), this unfortunately turned out not to be the case. There was also no opportunity for the chairs to open the Industry track, present some statistics about the track, and thank authors and reviewers (which is in part why I wrote this post).

If future chairs want our material, we archived everything, which hopefully makes chairing the track in the future much less work. A big thanks to our PC and Andrew Bagel, chair of the 2019 edition of the FSE Industry track, who provided us with valuable insights.

My specifications seemed (to me) modest: The hub should support 100W charging (since the 2019 MacBook Pro needs it), HDMI (for ergonomic reasons, I like to stay perfectly centered in front of my display, so I only need one video out; if you need two, or worse, two at 4K, welcome to adapter hell!), Ethernet, and a couple of type-A USB ports so I can plug-in my wireless mouse, keyboard, and other equipment.

When I started my quest, I was quite delighted to see that from around 20$, one could get USB-C hubs that effectively mirror the capabilities of docking stations of old. Something that in the pre-USB 3 era would have cost upward of 200 bucks, was now finally within reach: plug in one cable at work your desk and be done with it. Moreover, most of these devices are small, stylish aluminum devices that resemble the MacBook’s elegance perfectly.

It is astonishingly difficult to get a USB-C adapter hub that just works

Disappointment settled in quickly when I tried to sift-through the dozens of near-identical hubs on Amazon, many of which seem to have fake reviews: how does on even make a decision among those end- and nameless brands? So, I just started ordering adapters with the highest rating I could find, half-cosher reviews, and only the features I needed (I figured if they had more at the same price, that would naturally be detrimental to the performance of the adapter).

AUKEY USB C Hub 8 in 1, 4.5 star rating (42.99 €)

I bought the AUKEY USB C Hub 8 in 1, since it explicitly mentioned its 100 Watt and MacBook Pro compatibility. Big fail. The adapter not only cut power delivery to the MacBook every 30 or so seconds (complete with annoying MacOS sounds notifying you of the same), it also got hot enough to melt a near-by chocolate bar. Moreover, it had a high static. Lastly, its electromagnetic shielding was so poor that the only way I could get my Logitech nano receiver to work was by placing it on a leash-like USB extension cable. Otherwise, the cursor would uncontrollably jump from one corner to the other. I don’t know whether the adapter caused WIFI performance issues, but given its unshielded nature, I suspect so.

YXwin USB C Hub, 6 in 1 Type C USB C Adapter, 4.5 star rating (34.99 €)

The ominously named YXwin USB C Hub was also a fail, but on a more subtle level. At first, it seemed to do quite well: Power delivery was constant and the hub did not become too hot. Of course, electromagnetic shielding was also poor (or non-existent) so I couldn’t get my nano receiver off its leash just yet. But otherwise, I was quite happy and used it for a couple weeks in different environments. Until I noticed that my WIFI connection was slower and sometimes laggier than it should be. After two hours of playing around with the local WIFI installation, I finally zoomed-in on the culprit USB C hub. I found that the electromagnetic shielding of this hub was so bad that my WIFI’s speed (measured at 100 MBbit/s at the same spot with the peripheral-free MacBook) drop down to

• below 2 Mbit/s when the adapter was used with HDMI and power simultaneously
• a very jerky 15 Mbit/s when the adapter was used with HDMI.

VAVA USB Hub, 8-in-1 USB C Hub, 4.5 star rating (31.99 €, normally 39.99 €)

I just got this today. Everything works and it doesn’t make annoying noises. I don’t even need the USB leash, but can just directly plug-in my Logitech nano receiver, which indicates that the electromagnetic shielding is good. Even though there is a warning on the device that it can get extremely hot (130° Fahrenheit, that’s 54°C), it actually does not. Now here is a caveat: There are a few Amazon reviews of customers whose units failed after only a few months. Should this happen, I will update this blog post. If I have not, it still works happily.

Two conclusions: The cheapest product actually was best, and many Amazon reviews seem fake. Amazon should do something about it, as its reviews used to be an awesome resource to make customer-informed purchase decisions based on crowd wisdom. This is no wonder, as all hubs came with a more-or less snarky “thank you” note which promised longer warranty or other perks in exchange for a positive Amazon review.

Credits: Photo (CC) by Khalid Bencherifa

5 Artifacts to review is … 4 too many?

Reviewing an artifact means means to develop deep understanding of often years of work by a group of authors. I find it deeper than during a paper reviewer. Typically, one has to understand the paper, the artifact, their connection, and how to run it all. In practice, for ESEC/FSE 2020, with 5 papers, this was a Herculean task for me. It also required constant context switching since you have to work on multiple artifacts simultaneously to meet the tight deadline (10 days for reviewing).

Suggestion: Assign one or two artifacts per reviewer.

Interactive sessions cost time (and nerves)

For ESEC/FSE 2020, there was a one week period in which authors and reviewers conversed and tackled problems on GitHub. This makes evaluating artifacts much more cumbersome than reviewing a paper, which is mostly a one-off task you can freely schedule within at least a 2-week time frame. There is also no clear guidance on how fast one should come back to authors in case of a problem. I, for one, have the opinion that I should not be required to hot-patch code in the artifact with an editor. I still did because I wanted to help the authors. There is really no ethical norm on what constitutes good artifact reviewing and how to ensure all artifacts are treated equal. All my artifacts required multiple interactions with the authors to either get them to run or reproduce the results in the paper. Couple the work amount with the lack of prestige that reviewing for an artifact track has, and you end up with a combination that does not sound so attractive anymore.

Suggestion: Limit the number of interactions per artifact (i.e., authors are allowed 2 clarifications) or limit number of artifacts to review (see above).

Please have your local supercomputer at hand

0 out of the 5 artifacts I reviewed would have run on my laptop. Unfortunately, the system requirements were not always documented, so I partly found out after crashing my system or overheating the CPU. Not sure how to solve this for folks who do not have access to server-grade hardware? I imagine it is quite embarrassing to have to come back to the chairs after the artifact assignment and say “erm, sorry, I can actually not review any of these artifacts.” Another problem is that many artifacts require one to install arcane software or apply dubious system changes, which brings me to the next point:

Nothing works out of the box …

… except for the one artifact that used a Vagrant image! Is it good use of reviewers’ time to have to figure out an install process (and typically, a myriad of error messages the original authors had no chance of foreseeing)? I think not. A small minority of artifacts (e.g., benchmarks) might not want to use virtualization. All others (and this should really be 99%): Please, use an image. Note also that having an image does not make it harder to install the software elsewhere.

Suggestion: A virtual machine image should be mandatory. Considering the zoo of virtualization solutions (containers, images, app images), I would even prescribe a certain format for a given track. Exemptions from it should be rare and require explicit permission.

This convenient 154,000 line log file reproduces the results.

Oh, of course, one needs to parse it to get the aggregated results in Table 2, but never mind. Just figuring out what output has been produced where, and to which piece(s) in the paper it relates is often a Sisyphean task. Too many artifact documentations (which are generally of poor quality) end with the sentence “This produces the results.” Authors, please put yourself in the shoes of the reviewer. On the other extreme, there are artifacts that even produce LaTeX tables just like they appear in the paper. This is great per se, but at the same time, I need traceability.

Suggestion: There should be a step-by-step walk through on how to get the paper’s results.

Documentation chaos

Perhaps facilitated by mandating no less than five markdown documents, many artifacts did not adhere to the submission guidelines strictly. An abundance of information spread across a website, README, and wiki, can be helpful later on, but adds to the confusion of the reviewer.

Suggestion: Artifacts should have one canonical tutorial-style document for reviewers; additional information can be very helpful in practice and for later, but should be separate.

GitHub is not a good platform for reviewing

Look at the chaos in the screenshot, which is from the official ESEC/FSE artifact repository. As you can see, there are at least 5 different ways artifacts are stored there, and not all include the paper number. Such accidental complexity needs to be removed.

GitHub has other issues, too. GitHub is not great in

1. enforcing consistency, or communicating quickly to the chairs which badges to award.
2. getting an overview of the state of your artifacts.
3. staying anonymous. Re-using your anonymous GitHub reviewer id (as suggested in the reviewing instructions) lets one draw conclusions on who is who, by matching anonymous ids with previous real-name committees, which are openly accessible on the web. GitHub facilitates this by showing when someone joined the service, even when other activity is hidden.

And then there is always the nagging question of …

Wasn’t this just a bitcoin miner in disguise with a static printf buried somewhere that outputs the results?

Despite putting lots of time into the artifacts track (over 30h!), I would not be 100% confident in excluding this. Sure, I would be negatively surprised, but can I rule it out? Honest answer: no.

Suggestion: We need concrete guidelines for reviewers on how to verify an artifact – do I need to inspect the source code? Do I need to be able to build it myself? Do I need to mutate the input such as to observe mutated output (often times, verifying whether the output is still correct is non-trivial, due to step functions, etc.)? Do I need to verify all of the original results? Is a sample-based evaluation enough?